Keith E Morse1, Conner Brown2, Scott Fleming3, Irene Todd2, Austin Powell2, Alton Russell4, David Scheinker2, Scott M Sutherland5, Jonathan Lu3, Brendan Watkins2, Nigam H Shah3, Natalie M Pageler6,7, Jonathan P Palma8. 1. Division of Pediatric Hospital Medicine, Department of Pediatrics, Stanford University School of Medicine, Stanford, California, United States. 2. Information Services Department, Lucile Packard Children's Hospital, Stanford, Palo Alto, California, United States. 3. Department of Biomedical Data Science, Stanford University, Palo Alto, California, United States. 4. Harvard Medical School, Boston, Massachusetts, United States. 5. Division of Nephrology, Department of Pediatrics, Stanford University, Stanford, California, United States. 6. Division of Pediatric Critical Care Medicine, Department of Pediatrics, Stanford University School of Medicine, Stanford, California, United States. 7. Division of Systems Medicine, Department of Pediatrics, Stanford University School of Medicine, Stanford, California, United States. 8. Division of Neonatology, Department of Pediatrics, Orlando Health, Orlando, Florida, United States.
Abstract
OBJECTIVE: The purpose of this study is to evaluate the ability of three metrics to monitor for a reduction in performance of a chronic kidney disease (CKD) model deployed at a pediatric hospital. METHODS: The CKD risk model estimates a patient's risk of developing CKD 3 to 12 months following an inpatient admission. The model was developed on a retrospective dataset of 4,879 admissions from 2014 to 2018, then run silently on 1,270 admissions from April to October, 2019. Three metrics were used to monitor its performance during the silent phase: (1) standardized mean differences (SMDs); (2) performance of a "membership model"; and (3) response distribution analysis. Observed patient outcomes for the 1,270 admissions were used to calculate prospective model performance and the ability of the three metrics to detect performance changes. RESULTS: The deployed model had an area under the receiver-operator curve (AUROC) of 0.63 in the prospective evaluation, which was a significant decrease from an AUROC of 0.76 on retrospective data (p = 0.033). Among the three metrics, SMDs were significantly different for 66/75 (88%) of the model's input variables (p <0.05) between retrospective and deployment data. The membership model was able to discriminate between the two settings (AUROC = 0.71, p <0.0001) and the response distributions were significantly different (p <0.0001) for the two settings. CONCLUSION: This study suggests that the three metrics examined could provide early indication of performance deterioration in deployed models' performance. Thieme. All rights reserved.
OBJECTIVE: The purpose of this study is to evaluate the ability of three metrics to monitor for a reduction in performance of a chronic kidney disease (CKD) model deployed at a pediatric hospital. METHODS: The CKD risk model estimates a patient's risk of developing CKD 3 to 12 months following an inpatient admission. The model was developed on a retrospective dataset of 4,879 admissions from 2014 to 2018, then run silently on 1,270 admissions from April to October, 2019. Three metrics were used to monitor its performance during the silent phase: (1) standardized mean differences (SMDs); (2) performance of a "membership model"; and (3) response distribution analysis. Observed patient outcomes for the 1,270 admissions were used to calculate prospective model performance and the ability of the three metrics to detect performance changes. RESULTS: The deployed model had an area under the receiver-operator curve (AUROC) of 0.63 in the prospective evaluation, which was a significant decrease from an AUROC of 0.76 on retrospective data (p = 0.033). Among the three metrics, SMDs were significantly different for 66/75 (88%) of the model's input variables (p <0.05) between retrospective and deployment data. The membership model was able to discriminate between the two settings (AUROC = 0.71, p <0.0001) and the response distributions were significantly different (p <0.0001) for the two settings. CONCLUSION: This study suggests that the three metrics examined could provide early indication of performance deterioration in deployed models' performance. Thieme. All rights reserved.
Authors: Karel G M Moons; Andre Pascal Kengne; Diederick E Grobbee; Patrick Royston; Yvonne Vergouwe; Douglas G Altman; Mark Woodward Journal: Heart Date: 2012-03-07 Impact factor: 5.994
Authors: Robert Challen; Joshua Denny; Martin Pitt; Luke Gompels; Tom Edwards; Krasimira Tsaneva-Atanasova Journal: BMJ Qual Saf Date: 2019-01-12 Impact factor: 7.035