PURPOSE: Symptoms are vital outcomes for cancer clinical trials, observational research, and population-level surveillance. Patient-reported outcomes (PROs) are valuable for monitoring symptoms, yet there are many challenges to collecting PROs at scale. We sought to develop, test, and externally validate a deep learning model to extract symptoms from unstructured clinical notes in the electronic health record. METHODS: We randomly selected 1,225 outpatient progress notes from among patients treated at the Dana-Farber Cancer Institute between January 2016 and December 2019 and used 1,125 notes as our training/validation data set and 100 notes as our test data set. We evaluated the performance of 10 deep learning models for detecting 80 symptoms included in the National Cancer Institute's Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) framework. Model performance as compared with manual chart abstraction was assessed using standard metrics, and the highest performer was externally validated on a sample of 100 physician notes from a different clinical context. RESULTS: In our training and test data sets, 75 of the 80 candidate symptoms were identified. The ELECTRA-small model had the highest performance for symptom identification at the token level (ie, at the individual symptom level), with an F1 of 0.87 and a processing time of 3.95 seconds per note. For the 10 most common symptoms in the test data set, the F1 score ranged from 0.98 for anxious to 0.86 for fatigue. For external validation of the same symptoms, the note-level performance ranged from F1 = 0.97 for diarrhea and dizziness to F1 = 0.73 for swelling. CONCLUSION: Training a deep learning model to identify a wide range of electronic health record-documented symptoms relevant to cancer care is feasible. This approach could be used at the health system scale to complement to electronic PROs.
PURPOSE: Symptoms are vital outcomes for cancer clinical trials, observational research, and population-level surveillance. Patient-reported outcomes (PROs) are valuable for monitoring symptoms, yet there are many challenges to collecting PROs at scale. We sought to develop, test, and externally validate a deep learning model to extract symptoms from unstructured clinical notes in the electronic health record. METHODS: We randomly selected 1,225 outpatient progress notes from among patients treated at the Dana-Farber Cancer Institute between January 2016 and December 2019 and used 1,125 notes as our training/validation data set and 100 notes as our test data set. We evaluated the performance of 10 deep learning models for detecting 80 symptoms included in the National Cancer Institute's Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) framework. Model performance as compared with manual chart abstraction was assessed using standard metrics, and the highest performer was externally validated on a sample of 100 physician notes from a different clinical context. RESULTS: In our training and test data sets, 75 of the 80 candidate symptoms were identified. The ELECTRA-small model had the highest performance for symptom identification at the token level (ie, at the individual symptom level), with an F1 of 0.87 and a processing time of 3.95 seconds per note. For the 10 most common symptoms in the test data set, the F1 score ranged from 0.98 for anxious to 0.86 for fatigue. For external validation of the same symptoms, the note-level performance ranged from F1 = 0.97 for diarrhea and dizziness to F1 = 0.73 for swelling. CONCLUSION: Training a deep learning model to identify a wide range of electronic health record-documented symptoms relevant to cancer care is feasible. This approach could be used at the health system scale to complement to electronic PROs.
Authors: Anamaria R Yeung; Stephanie L Pugh; Ann H Klopp; Karen M Gil; Lari Wenzel; Shannon N Westin; David K Gaffney; William Small; Spencer Thompson; Desiree E Doncals; Guilherme H C Cantuaria; Brian P Yaremko; Amy Chang; Vijayananda Kundapur; Dasarahally S Mohan; Michael L Haas; Yong Bae Kim; Catherine L Ferguson; Snehal Deshmukh; Deborah W Bruner; Lisa A Kachnic Journal: J Clin Oncol Date: 2020-02-19 Impact factor: 44.544
Authors: Lev D Bubis; Laura Davis; Alyson Mahar; Lisa Barbera; Qing Li; Lesley Moody; Paul Karanicolas; Rinku Sutradhar; Natalie G Coburn Journal: J Clin Oncol Date: 2018-03-01 Impact factor: 44.544
Authors: Deborah Watkins Bruner; Laura J Hanisch; Bryce B Reeve; Andy M Trotti; Deborah Schrag; Laura Sit; Tito R Mendoza; Lori Minasian; Ann O'Mara; Andrea M Denicoff; Julia H Rowland; Michael Montello; Cindy Geoghegan; Amy P Abernethy; Steven B Clauser; Kathleen Castro; Sandra A Mitchell; Laurie Burke; Ann Marie Trentacosti; Ethan M Basch Journal: Transl Behav Med Date: 2011-03 Impact factor: 3.046
Authors: Suzanne Tamang; Manali I Patel; Douglas W Blayney; Julie Kuznetsov; Samuel G Finlayson; Yohan Vetteth; Nigam Shah Journal: J Oncol Pract Date: 2015-05 Impact factor: 3.840
Authors: Kenneth L Kehl; Wenxin Xu; Eva Lepisto; Haitham Elmarakeby; Michael J Hassett; Eliezer M Van Allen; Bruce E Johnson; Deborah Schrag Journal: JCO Clin Cancer Inform Date: 2020-08
Authors: Ethan Basch; Allison M Deal; Mark G Kris; Howard I Scher; Clifford A Hudis; Paul Sabbatini; Lauren Rogak; Antonia V Bennett; Amylou C Dueck; Thomas M Atkinson; Joanne F Chou; Dorothy Dulko; Laura Sit; Allison Barz; Paul Novotny; Michael Fruscione; Jeff A Sloan; Deborah Schrag Journal: J Clin Oncol Date: 2015-12-07 Impact factor: 44.544
Authors: Charlotta Lindvall; Chih-Ying Deng; Edward Moseley; Nicole Agaronnik; Areej El-Jawahri; Michael K Paasche-Orlow; Joshua R Lakin; Angelo Volandes; The Acp-Peace Investigators James A Tulsky Journal: J Pain Symptom Manage Date: 2021-07-14 Impact factor: 5.576