Hatem Al-Farra1,2, Bas A J M de Mol2, Anita C J Ravelli1, W J P P Ter Burg1, Saskia Houterman3, José P S Henriques2, Ameen Abu-Hanna1, M M Vis2, J Vos4, L Timmers5, W A L Tonino6, C E Schotborgh7, V Roolvink8, F Porta9, M G Stoel10, S Kats11, G Amoroso12, H W van der Werf13, P R Stella14, P de Jaegere15. 1. Department of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands. 2. Heart Center, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands. 3. Netherlands Heart Registration, Utrecht, the Netherlands. 4. Amphia Hospital, the Netherlands. 5. St. Antonius Hospital, the Netherlands. 6. Catharina Hospital, the Netherlands. 7. Haga Hospital, the Netherlands. 8. Isala, the Netherlands. 9. Leeuwarden Medical Center, the Netherlands. 10. Medisch Spectrum Twente, the Netherlands. 11. Maastricht University Medical Center, the Netherlands. 12. Onze Lieve Vrouwe Gasthuis, the Netherlands. 13. University Medical Center Groningen, the Netherlands. 14. University Medical Center Utrecht, the Netherlands. 15. Erasmus University Medical Center, the Netherlands.
Abstract
BACKGROUND: The predictive performance of the models FRANCE-2 and ACC-TAVI for early-mortality after Transcatheter Aortic Valve Implantation (TAVI) can decline over time and can be enhanced by updating them on new populations. We aim to update and internally and temporally validate these models using a recent TAVI-cohort from the Netherlands Heart Registration (NHR). METHODS: We used data of TAVI-patients treated in 2013-2017. For each original-model, the best update-method (model-intercept, model-recalibration, or model-revision) was selected by a closed-testing procedure. We internally validated both updated models with 1000 bootstrap samples. We also updated the models on the 2013-2016 dataset and temporally validated them on the 2017-dataset. Performance measures were the Area-Under ROC-curve (AU-ROC), Brier-score, and calibration graphs. RESULTS: We included 6177 TAVI-patients, with 4.5% observed early-mortality. The selected update-method for FRANCE-2 was model-intercept-update. Internal validation showed an AU-ROC of 0.63 (95%CI 0.62-0.66) and Brier-score of 0.04 (0.04-0.05). Calibration graphs show that it overestimates early-mortality. In temporal-validation, the AU-ROC was 0.61 (0.53-0.67).The selected update-method for ACC-TAVI was model-revision. In internal-validation, the AU-ROC was 0.63 (0.63-0.66) and Brier-score was 0.04 (0.04-0.05). The updated ACC-TAVI calibrates well up to a probability of 20%, and subsequently underestimates early-mortality. In temporal-validation the AU-ROC was 0.65 (0.58-0.72). CONCLUSION: Internal-validation of the updated models FRANCE-2 and ACC-TAVI with data from the NHR demonstrated improved performance, which was better than in external-validation studies and comparable to the original studies. In temporal-validation, ACC-TAVI outperformed FRANCE-2 because it suffered less from changes over time.
BACKGROUND: The predictive performance of the models FRANCE-2 and ACC-TAVI for early-mortality after Transcatheter Aortic Valve Implantation (TAVI) can decline over time and can be enhanced by updating them on new populations. We aim to update and internally and temporally validate these models using a recent TAVI-cohort from the Netherlands Heart Registration (NHR). METHODS: We used data of TAVI-patients treated in 2013-2017. For each original-model, the best update-method (model-intercept, model-recalibration, or model-revision) was selected by a closed-testing procedure. We internally validated both updated models with 1000 bootstrap samples. We also updated the models on the 2013-2016 dataset and temporally validated them on the 2017-dataset. Performance measures were the Area-Under ROC-curve (AU-ROC), Brier-score, and calibration graphs. RESULTS: We included 6177 TAVI-patients, with 4.5% observed early-mortality. The selected update-method for FRANCE-2 was model-intercept-update. Internal validation showed an AU-ROC of 0.63 (95%CI 0.62-0.66) and Brier-score of 0.04 (0.04-0.05). Calibration graphs show that it overestimates early-mortality. In temporal-validation, the AU-ROC was 0.61 (0.53-0.67).The selected update-method for ACC-TAVI was model-revision. In internal-validation, the AU-ROC was 0.63 (0.63-0.66) and Brier-score was 0.04 (0.04-0.05). The updated ACC-TAVI calibrates well up to a probability of 20%, and subsequently underestimates early-mortality. In temporal-validation the AU-ROC was 0.65 (0.58-0.72). CONCLUSION: Internal-validation of the updated models FRANCE-2 and ACC-TAVI with data from the NHR demonstrated improved performance, which was better than in external-validation studies and comparable to the original studies. In temporal-validation, ACC-TAVI outperformed FRANCE-2 because it suffered less from changes over time.
Keywords:
ACC-TAVI (ACC TVT), American College of Cardiology Transcatheter Valve Therapy; AU-PRC, Area Under the Precision-Recall Curve; AU-ROC, Area Under the Receiver Operating-Characteristic Curve; Amsterdam UMC, Amsterdam University Medical Center - location AMC (Academic Medical Center); BSS, Brier-skill score; Closed-testing procedure; EuroSCORE, European System for Cardiac Operative Risk Evaluation; External Validation; FRANCE-2, French Aortic National CoreValve and Edwards [15]; LVEF, Left Ventricular Ejection Fraction; MPM, Mortality Prediction Models; Model recalibration; Model updating; NHR, Netherlands Heart Registration (“Nederlandse Hart Registratie in Dutch”); NYHA, New York Heart Association; Prediction models; SAVR, Surgical Aortic Valve Replacement; TAVI (TAVR), Transcatheter Aortic Valve Implantation (Replacement); Transcatheter Aortic Valve Implantation (TAVI)
Authors: Samer A M Nashef; François Roques; Linda D Sharples; Johan Nilsson; Christopher Smith; Antony R Goldstone; Ulf Lockowandt Journal: Eur J Cardiothorac Surg Date: 2012-02-29 Impact factor: 4.191
Authors: Lucas Lodi-Junqueira; José L P da Silva; Lorena R Ferreira; Humberto L Gonçalves; Guilherme R S Athayde; Thalles O Gomes; Júlio C Borges; Bruno R Nascimento; Pedro A Lemos; Antônio L P Ribeiro Journal: Catheter Cardiovasc Interv Date: 2015-07-02 Impact factor: 2.692
Authors: Yvonne Vergouwe; Daan Nieboer; Rianne Oostenbrink; Thomas P A Debray; Gordon D Murray; Michael W Kattan; Hendrik Koffijberg; Karel G M Moons; Ewout W Steyerberg Journal: Stat Med Date: 2016-11-28 Impact factor: 2.373
Authors: Manuel Carnero-Alcázar; Luis Carlos Maroto; Javier Cobiella-Carnicer; Isidre Vilacosta; Luis Nombela-Franco; Ali Alswies; Enrique Villagrán-Medinilla; Carlos Macaya Journal: Eur J Cardiothorac Surg Date: 2017-04-01 Impact factor: 4.191
Authors: Alec Vahanian; Ottavio Alfieri; Felicita Andreotti; Manuel J Antunes; Gonzalo Barón-Esquivias; Helmut Baumgartner; Michael Andrew Borger; Thierry P Carrel; Michele De Bonis; Arturo Evangelista; Volkmar Falk; Bernard Lung; Patrizio Lancellotti; Luc Pierard; Susanna Price; Hans-Joachim Schäfers; Gerhard Schuler; Janina Stepinska; Karl Swedberg; Johanna Takkenberg; Ulrich Otto Von Oppell; Stephan Windecker; Jose Luis Zamorano; Marian Zembala Journal: Eur J Cardiothorac Surg Date: 2012-08-25 Impact factor: 4.191
Authors: Ben Van Calster; Kirsten Van Hoorde; Yvonne Vergouwe; Shabnam Bobdiwala; George Condous; Emma Kirk; Tom Bourne; Ewout W Steyerberg Journal: Diagn Progn Res Date: 2017-02-08
Authors: Maarten van Smeden; Georg Heinze; Ben Van Calster; Folkert W Asselbergs; Panos E Vardas; Nico Bruining; Peter de Jaegere; Jason H Moore; Spiros Denaxas; Anne Laure Boulesteix; Karel G M Moons Journal: Eur Heart J Date: 2022-08-14 Impact factor: 35.855