BACKGROUND: Optimal cutoff values for tests results involving continuous variables are often derived in a data-driven way. This approach, however, may lead to overly optimistic measures of diagnostic accuracy. We evaluated the magnitude of the bias in sensitivity and specificity associated with data-driven selection of cutoff values and examined potential solutions to reduce this bias. METHODS: Different sample sizes, distributions, and prevalences were used in a simulation study. We compared data-driven estimates of accuracy based on the Youden index with the true values and calculated the median bias. Three alternative approaches (assuming a specific distribution, leave-one-out, smoothed ROC curve) were examined for their ability to reduce this bias. RESULTS: The magnitude of bias caused by data-driven optimization of cutoff values was inversely related to sample size. If the true values for sensitivity and specificity are both 84%, the estimates in studies with a sample size of 40 will be approximately 90%. If the sample size increases to 200, the estimates will be 86%. The distribution of the test results had little impact on the amount of bias when sample size was held constant. More robust methods of optimizing cutoff values were less prone to bias, but the performance deteriorated if the underlying assumptions were not met. CONCLUSIONS: Data-driven selection of the optimal cutoff value can lead to overly optimistic estimates of sensitivity and specificity, especially in small studies. Alternative methods can reduce this bias, but finding robust estimates for cutoff values and accuracy requires considerable sample sizes.
BACKGROUND: Optimal cutoff values for tests results involving continuous variables are often derived in a data-driven way. This approach, however, may lead to overly optimistic measures of diagnostic accuracy. We evaluated the magnitude of the bias in sensitivity and specificity associated with data-driven selection of cutoff values and examined potential solutions to reduce this bias. METHODS: Different sample sizes, distributions, and prevalences were used in a simulation study. We compared data-driven estimates of accuracy based on the Youden index with the true values and calculated the median bias. Three alternative approaches (assuming a specific distribution, leave-one-out, smoothed ROC curve) were examined for their ability to reduce this bias. RESULTS: The magnitude of bias caused by data-driven optimization of cutoff values was inversely related to sample size. If the true values for sensitivity and specificity are both 84%, the estimates in studies with a sample size of 40 will be approximately 90%. If the sample size increases to 200, the estimates will be 86%. The distribution of the test results had little impact on the amount of bias when sample size was held constant. More robust methods of optimizing cutoff values were less prone to bias, but the performance deteriorated if the underlying assumptions were not met. CONCLUSIONS: Data-driven selection of the optimal cutoff value can lead to overly optimistic estimates of sensitivity and specificity, especially in small studies. Alternative methods can reduce this bias, but finding robust estimates for cutoff values and accuracy requires considerable sample sizes.
Authors: Anna Luíza Damaceno Araújo; Lady Paola Aristizábal Arboleda; Natalia Rangel Palmier; Jéssica Montenegro Fonsêca; Mariana de Pauli Paglioni; Wagner Gomes-Silva; Ana Carolina Prado Ribeiro; Thaís Bianca Brandão; Luciana Estevam Simonato; Paul M Speight; Felipe Paiva Fonseca; Marcio Ajudarte Lopes; Oslei Paes de Almeida; Pablo Agustin Vargas; Cristhian Camilo Madrid Troconis; Alan Roger Santos-Silva Journal: Virchows Arch Date: 2019-01-26 Impact factor: 4.064
Authors: Patrick J Bolan; Eunhee Kim; Benjamin A Herman; Gillian M Newstead; Mark A Rosen; Mitchell D Schnall; Etta D Pisano; Paul T Weatherall; Elizabeth A Morris; Constance D Lehman; Michael Garwood; Michael T Nelson; Douglas Yee; Sandra M Polin; Laura J Esserman; Constantine A Gatsonis; Gregory J Metzger; David C Newitt; Savannah C Partridge; Nola M Hylton Journal: J Magn Reson Imaging Date: 2016-12-16 Impact factor: 4.813
Authors: Mariska M G Leeflang; Jonathan J Deeks; Constantine Gatsonis; Patrick M M Bossuyt Journal: Ann Intern Med Date: 2008-12-16 Impact factor: 25.391
Authors: Kimiko A Broeze; Brent C Opmeer; Lucas M Bachmann; Frank J Broekmans; Patrick M M Bossuyt; Sjors F P J Coppus; Neil P Johnson; Khalid S Khan; Gerben ter Riet; Fulco van der Veen; Madelon van Wely; Ben W J Mol Journal: BMC Med Res Methodol Date: 2009-03-27 Impact factor: 4.615