Christopher R Manz1,2,3,4, Jinbo Chen5, Manqing Liu4, Corey Chivers6, Susan Harkness Regli6, Jennifer Braun3, Michael Draugelis6, C William Hanson6, Lawrence N Shulman1,2,3, Lynn M Schuchter1,2,3, Nina O'Connor1, Justin E Bekelman1,2,3,4,7, Mitesh S Patel1,2,4,7,8, Ravi B Parikh1,2,3,4,7,8. 1. Department of Medicine, University of Pennsylvania Perelman School of Medicine, Philadelphia. 2. Penn Center for Cancer Care Innovation, University of Pennsylvania, Philadelphia. 3. Abramson Cancer Center, University of Pennsylvania, Philadelphia. 4. Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia. 5. Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania Perelman School of Medicine, Philadelphia. 6. Penn Medicine, University of Pennsylvania, Philadelphia. 7. Department of Medical Ethics and Health Policy, University of Pennsylvania Perelman School of Medicine, Philadelphia. 8. Corporal Michael J. Crescenz VA Medical Center, Philadelphia, Pennsylvania.
Abstract
IMPORTANCE: Machine learning (ML) algorithms can identify patients with cancer at risk of short-term mortality to inform treatment and advance care planning. However, no ML mortality risk prediction algorithm has been prospectively validated in oncology or compared with routinely used prognostic indices. OBJECTIVE: To validate an electronic health record-embedded ML algorithm that generated real-time predictions of 180-day mortality risk in a general oncology cohort. DESIGN, SETTING, AND PARTICIPANTS: This prognostic study comprised a prospective cohort of patients with outpatient oncology encounters between March 1, 2019, and April 30, 2019. An ML algorithm, trained on retrospective data from a subset of practices, predicted 180-day mortality risk between 4 and 8 days before a patient's encounter. Patient encounters took place in 18 medical or gynecologic oncology practices, including 1 tertiary practice and 17 general oncology practices, within a large US academic health care system. Patients aged 18 years or older with outpatient oncology or hematology and oncology encounters were included in the analysis. Patients were excluded if their appointment was scheduled after weekly predictions were generated and if they were only evaluated in benign hematology, palliative care, or rehabilitation practices. EXPOSURES: Gradient-boosting ML binary classifier. MAIN OUTCOMES AND MEASURES: The primary outcome was the patients' 180-day mortality from the index encounter. The primary performance metric was the area under the receiver operating characteristic curve (AUC). RESULTS: Among 24 582 patients, 1022 (4.2%) died within 180 days of their index encounter. Their median (interquartile range) age was 64.6 (53.6-73.2) years, 15 319 (62.3%) were women, 18 015 (76.0%) were White, and 10 658 (43.4%) were seen in the tertiary practice. The AUC was 0.89 (95% CI, 0.88-0.90) for the full cohort. The AUC varied across disease-specific groups within the tertiary practice (AUC ranging from 0.74 to 0.96) but was similar between the tertiary and general oncology practices. At a prespecified 40% mortality risk threshold used to differentiate high- vs low-risk patients, observed 180-day mortality was 45.2% (95% CI, 41.3%-49.1%) in the high-risk group vs 3.1% (95% CI, 2.9%-3.3%) in the low-risk group. Integrating the algorithm into the Eastern Cooperative Oncology Group and Elixhauser comorbidity index-based classifiers resulted in favorable reclassification (net reclassification index, 0.09 [95% CI, 0.04-0.14] and 0.23 [95% CI, 0.20-0.27], respectively). CONCLUSIONS AND RELEVANCE: In this prognostic study, an ML algorithm was feasibly integrated into the electronic health record to generate real-time, accurate predictions of short-term mortality for patients with cancer and outperformed routinely used prognostic indices. This algorithm may be used to inform behavioral interventions and prompt earlier conversations about goals of care and end-of-life preferences among patients with cancer.
IMPORTANCE: Machine learning (ML) algorithms can identify patients with cancer at risk of short-term mortality to inform treatment and advance care planning. However, no ML mortality risk prediction algorithm has been prospectively validated in oncology or compared with routinely used prognostic indices. OBJECTIVE: To validate an electronic health record-embedded ML algorithm that generated real-time predictions of 180-day mortality risk in a general oncology cohort. DESIGN, SETTING, AND PARTICIPANTS: This prognostic study comprised a prospective cohort of patients with outpatient oncology encounters between March 1, 2019, and April 30, 2019. An ML algorithm, trained on retrospective data from a subset of practices, predicted 180-day mortality risk between 4 and 8 days before a patient's encounter. Patient encounters took place in 18 medical or gynecologic oncology practices, including 1 tertiary practice and 17 general oncology practices, within a large US academic health care system. Patients aged 18 years or older with outpatient oncology or hematology and oncology encounters were included in the analysis. Patients were excluded if their appointment was scheduled after weekly predictions were generated and if they were only evaluated in benign hematology, palliative care, or rehabilitation practices. EXPOSURES: Gradient-boosting ML binary classifier. MAIN OUTCOMES AND MEASURES: The primary outcome was the patients' 180-day mortality from the index encounter. The primary performance metric was the area under the receiver operating characteristic curve (AUC). RESULTS: Among 24 582 patients, 1022 (4.2%) died within 180 days of their index encounter. Their median (interquartile range) age was 64.6 (53.6-73.2) years, 15 319 (62.3%) were women, 18 015 (76.0%) were White, and 10 658 (43.4%) were seen in the tertiary practice. The AUC was 0.89 (95% CI, 0.88-0.90) for the full cohort. The AUC varied across disease-specific groups within the tertiary practice (AUC ranging from 0.74 to 0.96) but was similar between the tertiary and general oncology practices. At a prespecified 40% mortality risk threshold used to differentiate high- vs low-risk patients, observed 180-day mortality was 45.2% (95% CI, 41.3%-49.1%) in the high-risk group vs 3.1% (95% CI, 2.9%-3.3%) in the low-risk group. Integrating the algorithm into the Eastern Cooperative Oncology Group and Elixhauser comorbidity index-based classifiers resulted in favorable reclassification (net reclassification index, 0.09 [95% CI, 0.04-0.14] and 0.23 [95% CI, 0.20-0.27], respectively). CONCLUSIONS AND RELEVANCE: In this prognostic study, an ML algorithm was feasibly integrated into the electronic health record to generate real-time, accurate predictions of short-term mortality for patients with cancer and outperformed routinely used prognostic indices. This algorithm may be used to inform behavioral interventions and prompt earlier conversations about goals of care and end-of-life preferences among patients with cancer.
Authors: Brian J Douthit; Rachel L Walden; Kenrick Cato; Cynthia P Coviak; Christopher Cruz; Fabio D'Agostino; Thompson Forbes; Grace Gao; Theresa A Kapetanovic; Mikyoung A Lee; Lisiane Pruinelli; Mary A Schultz; Ann Wieben; Alvin D Jeffery Journal: Appl Clin Inform Date: 2022-02-09 Impact factor: 2.342
Authors: Ravi B Parikh; Christopher R Manz; Maria N Nelson; Chalanda N Evans; Susan H Regli; Nina O'Connor; Lynn M Schuchter; Lawrence N Shulman; Mitesh S Patel; Joanna Paladino; Judy A Shea Journal: Support Care Cancer Date: 2022-01-30 Impact factor: 3.603
Authors: Charles Faselis; Joel A Nations; Charity J Morgan; Jared Antevil; Jeffrey M Roseman; Sijian Zhang; Gregg C Fonarow; Helen M Sheriff; Gregory D Trachiotis; Richard M Allman; Prakash Deedwania; Qing Zeng-Trietler; Daniel D Taub; Amiya A Ahmed; George Howard; Ali Ahmed Journal: JAMA Oncol Date: 2022-10-01 Impact factor: 33.006
Authors: Michael F Gensheimer; Sonya Aggarwal; Kathryn R K Benson; Justin N Carter; A Solomon Henry; Douglas J Wood; Scott G Soltys; Steven Hancock; Erqi Pollom; Nigam H Shah; Daniel T Chang Journal: J Am Med Inform Assoc Date: 2021-06-12 Impact factor: 4.497
Authors: Ajeet Gajra; Marjorie E Zettler; Kelly A Miller; John G Frownfelter; John Showalter; Amy W Valley; Sanya Sharma; Shreenath Sridharan; Jonathan K Kish; Sibel Blau Journal: JCO Oncol Pract Date: 2021-09-10