Akshay Swaminathan1, Rockli Kim2, S V Subramanian3. 1. Quantitative Sciences, Flatiron Health, New York, NY. 2. Division of Health Policy and Management, College of Health Science, Korea University, Seoul, South Korea; Harvard Center for Population & Development Studies, Cambridge, MA. 3. Harvard Center for Population & Development Studies, Cambridge, MA; Department of Social and Behavioral Sciences, Harvard T.H. Chan School of Public Health, Boston, MA. Electronic address: svsubram@hsph.harvard.edu.
Abstract
PURPOSE: Epidemiologic studies often conflate the strength of association with predictive accuracy and build classification models based on arbitrarily selected probability cutoffs without considering the cost of misclassification. We illustrated these common pitfalls by building association, prediction, and classification models using birthweight as an exposure and child mortality and child anthropometric failure as outcomes. METHODS: Nationally representative samples of 188,819 and 164,113 children aged less than 5 years across India were used for our analysis of mortality and anthropometric failure, respectively. We assessed outcomes of neonatal, postneonatal, and child mortality as well as stunting, wasting, and underweight. Birthweight was the main exposure. We used adjusted and unadjusted logistic regression models to evaluate association strength, univariable and multivariable logistic regression models trained on 80% of the data using 10-fold cross-validation to evaluate predictive power, and classification models across a series of possible misclassification cost scenarios to evaluate classification accuracy. RESULTS: Birthweight was strongly associated with five of six outcomes (P < .001), and associations were robust to covariate adjustment. Prediction models evaluated on the test set showed that birthweight was a poor discriminator of all outcomes (area under the curve < 0.62), and that adding birthweight to a multivariable model did not meaningfully improve discrimination. Prediction models for anthropometric failure showed substantially better calibration than prediction models for mortality. Depending on the ratio of false positive (FP) cost to false negative (FN) cost, the probability cutoff that minimized total misclassification cost ranged from 0.116 (cost ratio = 7:93) to 0.706 (cost ratio = 4:1), TPR ranged from 0.999 to 0.004, and PPV ranged from 0.355 to 0.867.. CONCLUSIONS: Although birthweight is strongly associated with mortality and anthropometric failure, it is a poor predictor of child health outcomes, highlighting that strong associations do not imply predictive power. We recommend that (1) future research focus on building predictive models for anthropometric failure given their clinical relevance in diagnosing individual cases, and that (2) studies that build classifiers report performance metrics across a range of cutoffs to account for variation in the cost of FPs and FNs.
PURPOSE: Epidemiologic studies often conflate the strength of association with predictive accuracy and build classification models based on arbitrarily selected probability cutoffs without considering the cost of misclassification. We illustrated these common pitfalls by building association, prediction, and classification models using birthweight as an exposure and childmortality and child anthropometric failure as outcomes. METHODS: Nationally representative samples of 188,819 and 164,113 children aged less than 5 years across India were used for our analysis of mortality and anthropometric failure, respectively. We assessed outcomes of neonatal, postneonatal, and childmortality as well as stunting, wasting, and underweight. Birthweight was the main exposure. We used adjusted and unadjusted logistic regression models to evaluate association strength, univariable and multivariable logistic regression models trained on 80% of the data using 10-fold cross-validation to evaluate predictive power, and classification models across a series of possible misclassification cost scenarios to evaluate classification accuracy. RESULTS: Birthweight was strongly associated with five of six outcomes (P < .001), and associations were robust to covariate adjustment. Prediction models evaluated on the test set showed that birthweight was a poor discriminator of all outcomes (area under the curve < 0.62), and that adding birthweight to a multivariable model did not meaningfully improve discrimination. Prediction models for anthropometric failure showed substantially better calibration than prediction models for mortality. Depending on the ratio of false positive (FP) cost to false negative (FN) cost, the probability cutoff that minimized total misclassification cost ranged from 0.116 (cost ratio = 7:93) to 0.706 (cost ratio = 4:1), TPR ranged from 0.999 to 0.004, and PPV ranged from 0.355 to 0.867.. CONCLUSIONS: Although birthweight is strongly associated with mortality and anthropometric failure, it is a poor predictor of child health outcomes, highlighting that strong associations do not imply predictive power. We recommend that (1) future research focus on building predictive models for anthropometric failure given their clinical relevance in diagnosing individual cases, and that (2) studies that build classifiers report performance metrics across a range of cutoffs to account for variation in the cost of FPs and FNs.