Robert Y Lee1, Lyndia C Brumback2, William B Lober3, James Sibley3, Elizabeth L Nielsen1, Patsy D Treece4, Erin K Kross1, Elizabeth T Loggers5, James A Fausto6, Charlotta Lindvall7, Ruth A Engelberg1, J Randall Curtis8. 1. Cambia Palliative Care Center of Excellence, University of Washington, Seattle, Washington, USA; Division of Pulmonary, Critical Care, and Sleep Medicine, Department of Medicine, Harborview Medical Center, University of Washington, Seattle, Washington, USA. 2. Cambia Palliative Care Center of Excellence, University of Washington, Seattle, Washington, USA; Department of Biostatistics, University of Washington, Seattle, Washington, USA. 3. Cambia Palliative Care Center of Excellence, University of Washington, Seattle, Washington, USA; Department of Biobehavioral Nursing and Health Informatics, University of Washington, Seattle, Washington, USA; Department of Bioinformatics and Medical Education, University of Washington, Seattle, Washington, USA. 4. Cambia Palliative Care Center of Excellence, University of Washington, Seattle, Washington, USA; Division of Pulmonary, Critical Care, and Sleep Medicine, Department of Medicine, Harborview Medical Center, University of Washington, Seattle, Washington, USA; Department of Biobehavioral Nursing and Health Informatics, University of Washington, Seattle, Washington, USA. 5. Cambia Palliative Care Center of Excellence, University of Washington, Seattle, Washington, USA; Clinical Research Division, Fred Hutchinson Cancer Research Center, Seattle, Washington, USA; Seattle Cancer Care Alliance, Seattle, Washington, USA. 6. Cambia Palliative Care Center of Excellence, University of Washington, Seattle, Washington, USA; Department of Family Medicine, University of Washington, Seattle, Washington, USA. 7. Department of Psychosocial Oncology and Palliative Care, Dana-Farber Cancer Institute, Boston, Massachusetts, USA. 8. Cambia Palliative Care Center of Excellence, University of Washington, Seattle, Washington, USA; Division of Pulmonary, Critical Care, and Sleep Medicine, Department of Medicine, Harborview Medical Center, University of Washington, Seattle, Washington, USA; Department of Biobehavioral Nursing and Health Informatics, University of Washington, Seattle, Washington, USA; Department of Bioethics and Humanities, University of Washington, Seattle, Washington, USA. Electronic address: jrc@u.washington.edu.
Abstract
CONTEXT: Goals-of-care discussions are an important quality metric in palliative care. However, goals-of-care discussions are often documented as free text in diverse locations. It is difficult to identify these discussions in the electronic health record (EHR) efficiently. OBJECTIVES: To develop, train, and test an automated approach to identifying goals-of-care discussions in the EHR, using natural language processing (NLP) and machine learning (ML). METHODS: From the electronic health records of an academic health system, we collected a purposive sample of 3183 EHR notes (1435 inpatient notes and 1748 outpatient notes) from 1426 patients with serious illness over 2008-2016, and manually reviewed each note for documentation of goals-of-care discussions. Separately, we developed a program to identify notes containing documentation of goals-of-care discussions using NLP and supervised ML. We estimated the performance characteristics of the NLP/ML program across 100 pairs of randomly partitioned training and test sets. We repeated these methods for inpatient-only and outpatient-only subsets. RESULTS: Of 3183 notes, 689 contained documentation of goals-of-care discussions. The mean sensitivity of the NLP/ML program was 82.3% (SD 3.2%), and the mean specificity was 97.4% (SD 0.7%). NLP/ML results had a median positive likelihood ratio of 32.2 (IQR 27.5-39.2) and a median negative likelihood ratio of 0.18 (IQR 0.16-0.20). Performance was better in inpatient-only samples than outpatient-only samples. CONCLUSION: Using NLP and ML techniques, we developed a novel approach to identifying goals-of-care discussions in the EHR. NLP and ML represent a potential approach toward measuring goals-of-care discussions as a research outcome and quality metric.
CONTEXT: Goals-of-care discussions are an important quality metric in palliative care. However, goals-of-care discussions are often documented as free text in diverse locations. It is difficult to identify these discussions in the electronic health record (EHR) efficiently. OBJECTIVES: To develop, train, and test an automated approach to identifying goals-of-care discussions in the EHR, using natural language processing (NLP) and machine learning (ML). METHODS: From the electronic health records of an academic health system, we collected a purposive sample of 3183 EHR notes (1435 inpatient notes and 1748 outpatient notes) from 1426 patients with serious illness over 2008-2016, and manually reviewed each note for documentation of goals-of-care discussions. Separately, we developed a program to identify notes containing documentation of goals-of-care discussions using NLP and supervised ML. We estimated the performance characteristics of the NLP/ML program across 100 pairs of randomly partitioned training and test sets. We repeated these methods for inpatient-only and outpatient-only subsets. RESULTS: Of 3183 notes, 689 contained documentation of goals-of-care discussions. The mean sensitivity of the NLP/ML program was 82.3% (SD 3.2%), and the mean specificity was 97.4% (SD 0.7%). NLP/ML results had a median positive likelihood ratio of 32.2 (IQR 27.5-39.2) and a median negative likelihood ratio of 0.18 (IQR 0.16-0.20). Performance was better in inpatient-only samples than outpatient-only samples. CONCLUSION: Using NLP and ML techniques, we developed a novel approach to identifying goals-of-care discussions in the EHR. NLP and ML represent a potential approach toward measuring goals-of-care discussions as a research outcome and quality metric.
Authors: Anahita Davoudi; Hegler Tissot; Abigail Doucette; Peter E Gabriel; Ravi Parikh; Danielle L Mowery; Stephen P Miranda Journal: AMIA Annu Symp Proc Date: 2022-05-23
Authors: Tamryn F Gray; Anne Kwok; Khuyen M Do; Sandra Zeng; Edward T Moseley; Yasser M Dbeis; Renato Umeton; James A Tulsky; Areej El-Jawahri; Charlotta Lindvall Journal: JMIR Med Inform Date: 2022-06-15
Authors: Alison M Uyeda; J Randall Curtis; Ruth A Engelberg; Lyndia C Brumback; Yue Guo; James Sibley; William B Lober; Trevor Cohen; Janaki Torrence; Joanna Heywood; Sudiptho R Paul; Erin K Kross; Robert Y Lee Journal: J Pain Symptom Manage Date: 2022-02-16 Impact factor: 5.576
Authors: Charlotta Lindvall; Chih-Ying Deng; Edward Moseley; Nicole Agaronnik; Areej El-Jawahri; Michael K Paasche-Orlow; Joshua R Lakin; Angelo Volandes; The Acp-Peace Investigators James A Tulsky Journal: J Pain Symptom Manage Date: 2021-07-14 Impact factor: 5.576