Giovanna Cilluffo1,2, Salvatore Fasola1,2, Giuliana Ferrante3, Laura Montalbano1, Ilaria Baiardini4, Luciana Indinnimeo5, Giovanni Viegi1,6, Joao A Fonseca7, Stefania La Grutta1. 1. Institute for Biomedical Research and Innovation, National Research Council of Italy, Palermo, Italy. 2. Department of Economical, Business and Statistical Science, University of Palermo, Palermo, Italy. 3. Department of Health Promotion Sciences, Maternal and Infant Care, Internal Medicine and Medical Specialities, University of Palermo, Italy. 4. Department of Biomedical Sciences, Humanitas University, Milan, Italy. 5. Department of Pediatrics and NPI, University of Roma Sapienza, Rome, Italy. 6. Institute of Clinical Physiology, Pulmonary Environmental Epidemiology Unit, National Research Council of Italy, Pisa, Italy. 7. Department of Immunoallergy, CUF Porto Hospital and Institute, Porto, Portugal.
Abstract
BACKGROUND: The use of receiver operating characteristic curves, or "ROC analysis," has become quite common in biomedical research to support decisions. However, sensitivity, specificity, and misclassification rates are still often estimated using the training sample, overlooking the risk of overrating the test performance. METHODS: A simulation study was performed to highlight the inferential implications of splitting (or not) the dataset into training and test set. The normality assumption was made for the classifier given the disease status, and the Youden's criterion considered for the detection of the optimal cutoff. Then, an ROC analysis with sample split was applied to assess the discriminant validity of the Italian version of the Control of Allergic Rhinitis and Asthma Test (CARATkids) questionnaire for children with asthma and rhinitis, for which recent studies may have reported liberal performance estimates. RESULTS: The simulation study showed that both single split and cross-validation (CV) provided unbiased estimators of sensitivity, specificity, and misclassification rate, therefore allowing computation of confidence intervals. For the Italian CARATkids questionnaire, the misclassification rate estimated by fivefold CV was 0.22, with 95% confidence interval 0.14 to 0.30, indicating an acceptable discriminant validity. CONCLUSIONS: Splitting into training and test set avoids overrating the test performance in ROC analysis. Validated through this method, the Italian CARATkids is valid for assessing disease control in children with asthma and rhinitis. Georg Thieme Verlag KG Stuttgart · New York.
BACKGROUND: The use of receiver operating characteristic curves, or "ROC analysis," has become quite common in biomedical research to support decisions. However, sensitivity, specificity, and misclassification rates are still often estimated using the training sample, overlooking the risk of overrating the test performance. METHODS: A simulation study was performed to highlight the inferential implications of splitting (or not) the dataset into training and test set. The normality assumption was made for the classifier given the disease status, and the Youden's criterion considered for the detection of the optimal cutoff. Then, an ROC analysis with sample split was applied to assess the discriminant validity of the Italian version of the Control of Allergic Rhinitis and Asthma Test (CARATkids) questionnaire for children with asthma and rhinitis, for which recent studies may have reported liberal performance estimates. RESULTS: The simulation study showed that both single split and cross-validation (CV) provided unbiased estimators of sensitivity, specificity, and misclassification rate, therefore allowing computation of confidence intervals. For the Italian CARATkids questionnaire, the misclassification rate estimated by fivefold CV was 0.22, with 95% confidence interval 0.14 to 0.30, indicating an acceptable discriminant validity. CONCLUSIONS: Splitting into training and test set avoids overrating the test performance in ROC analysis. Validated through this method, the Italian CARATkids is valid for assessing disease control in children with asthma and rhinitis. Georg Thieme Verlag KG Stuttgart · New York.
Authors: Giovanna Cilluffo; Anna Maria Zicari; Giuliana Ferrante; Velia Malizia; Salvatore Fasola; Marzia Duse; Giovanna De Castro; Valentina De Vittori; Laura Schiavi; Giulia Brindisi; Paolo Petrelli; Stefania La Grutta Journal: BMC Med Res Methodol Date: 2020-04-17 Impact factor: 4.615