Amir Hadanny1, Roni Shouval2, Jianhua Wu3, Chris P Gale4, Ron Unger5, Doron Zahger6, Shmuel Gottlieb7, Shlomi Matetzky8, Ilan Goldenberg9, Roy Beigel10, Zaza Iakobishvili11. 1. The Mina and Everard Goodman Faculty of Life Sciences, Bar-Ilan University, Ramat-Gan, Israel; Sackler School of Medicine, Tel-Aviv University, Tel-Aviv, Israel. Electronic address: amir.had@gmail.com. 2. The Mina and Everard Goodman Faculty of Life Sciences, Bar-Ilan University, Ramat-Gan, Israel; Sackler School of Medicine, Tel-Aviv University, Tel-Aviv, Israel; Adult Bone Marrow Transplantation Service, Memorial Sloan Kettering Cancer Center, New York, NY, United States of America. 3. Leeds Institute for Data Analytics, University of Leeds, Leeds, United Kingdom. 4. Leeds Institute for Data Analytics, University of Leeds, Leeds, United Kingdom; Leeds Institute of Cardiovascular and Metabolic Medicine, School of Medicine, University of Leeds, Leeds, United Kingdom; Department of Cardiology, Leeds Teaching Hospitals NHS Trust, Leeds, United Kingdom. 5. The Mina and Everard Goodman Faculty of Life Sciences, Bar-Ilan University, Ramat-Gan, Israel. 6. Department of Cardiology, Soroka University Medical Center, Faculty of Health Sciences, Ben Gurion University of the Negev, Beer Sheva, Israel. 7. Sackler School of Medicine, Tel-Aviv University, Tel-Aviv, Israel; Jesselson Integrated Heart Center, Shaare Zedek Medical Center, Jerusalem, Israel. 8. Sackler School of Medicine, Tel-Aviv University, Tel-Aviv, Israel; The Heart Institute, Sheba Medical Center, Tel Hashomer, Israel. 9. Sackler School of Medicine, Tel-Aviv University, Tel-Aviv, Israel; The Heart Institute, Sheba Medical Center, Tel Hashomer, Israel; Israeli Association for Cardiovascular Trials, Sheba Medical Center, Tel Hashomer, Israel. 10. Sackler School of Medicine, Tel-Aviv University, Tel-Aviv, Israel; Department of Cardiology, The Cardiovascular Division, Sheba Medical Center, Tel Hashomer, Israel. 11. Department of Cardiology, Soroka University Medical Center, Faculty of Health Sciences, Ben Gurion University of the Negev, Beer Sheva, Israel; Department of Cardiology, Tel Aviv Jaffa district, Clalit Health Services, Israel; Department of Cardiology, Samson Assuta Ashdod Hospital, Israel.
Abstract
BACKGROUND: Clinical risk assessment with quantitative formal risk scores may add to intuitive physician risk assessment and are advised by the international guidelines for the management of acute coronary syndrome (ACS) patients. Most previous studies have used the binary regression/classification approach (dead/alive) for long-term mortality post-ACS, without considering the time-to-event as in survival analysis. The use of machine learning (ML)-based survival models has yet to be validated. The primary objective was to compare survival prediction performance of 1-year mortality following ACS of two newly developed ML-based models [random survival forest (RSF) and deep learning (DeepSurv)] with the traditional Cox-proportional hazard (CPH) model. The secondary objective was external validation of the findings. METHODS: This was a retrospective, supervised learning data mining study based on the Acute Coronary Syndrome Israeli Survey (ACSIS) and the Myocardial Ischemia National Audit Project (MINAP). The ACSIS data were divided to train/test in a 70/30 fashion. Next, the models were externally validated on the MINAP data. Harrell's C-index, inverse probability of censoring weighting (IPCW), and the Brier-score were used for models' performance comparison. RESULTS: RSF performed best among the three models, with Harrell's C-index on training and testing sets reaching 0.953 and 0.924 respectively, followed by CPH multivariate selected model (0.805/0.849), CPH Univariate selected model (0.828/0.806), DeepSurv model (0.801/0.804), and the traditional CPH model (0.826/0.738). The RSF model also had the highest performance on the validation data set with 0.811 for Harrell's C-index, 0.844 for IPCW, and 0.093 for Brier score. The CPH model performance on the validation set had C-index range between 0.689 to 0.790, 0.713 to 0.826 for IPCW, and 0.094 to 0.103 Brier score. CONCLUSIONS: RSF survival predictions for long-term mortality post-ACS show improved model performance compared with the classic statistical method. This may benefit patients by allowing better risk stratification and tailored therapy, however further prospective evaluations are required.
BACKGROUND: Clinical risk assessment with quantitative formal risk scores may add to intuitive physician risk assessment and are advised by the international guidelines for the management of acute coronary syndrome (ACS) patients. Most previous studies have used the binary regression/classification approach (dead/alive) for long-term mortality post-ACS, without considering the time-to-event as in survival analysis. The use of machine learning (ML)-based survival models has yet to be validated. The primary objective was to compare survival prediction performance of 1-year mortality following ACS of two newly developed ML-based models [random survival forest (RSF) and deep learning (DeepSurv)] with the traditional Cox-proportional hazard (CPH) model. The secondary objective was external validation of the findings. METHODS: This was a retrospective, supervised learning data mining study based on the Acute Coronary Syndrome Israeli Survey (ACSIS) and the Myocardial Ischemia National Audit Project (MINAP). The ACSIS data were divided to train/test in a 70/30 fashion. Next, the models were externally validated on the MINAP data. Harrell's C-index, inverse probability of censoring weighting (IPCW), and the Brier-score were used for models' performance comparison. RESULTS: RSF performed best among the three models, with Harrell's C-index on training and testing sets reaching 0.953 and 0.924 respectively, followed by CPH multivariate selected model (0.805/0.849), CPH Univariate selected model (0.828/0.806), DeepSurv model (0.801/0.804), and the traditional CPH model (0.826/0.738). The RSF model also had the highest performance on the validation data set with 0.811 for Harrell's C-index, 0.844 for IPCW, and 0.093 for Brier score. The CPH model performance on the validation set had C-index range between 0.689 to 0.790, 0.713 to 0.826 for IPCW, and 0.094 to 0.103 Brier score. CONCLUSIONS: RSF survival predictions for long-term mortality post-ACS show improved model performance compared with the classic statistical method. This may benefit patients by allowing better risk stratification and tailored therapy, however further prospective evaluations are required.