Abhyuday Jagannatha1, Feifan Liu2, Weisong Liu3,4, Hong Yu5,6,7,8. 1. College of Information and Computer Sciences, University of Massachusetts, Amherst, MA, USA. 2. Department of Quantitative Health Sciences and Radiology, University of Massachusetts Medical School, Worcester, MA, USA. 3. Department of Computer Science, University of Massachusetts, 220 Pawtucket St., Lowell, MA, 01854-2874, USA. 4. Department of Medicine, University of Massachusetts Medical School, Worcester, MA, USA. 5. College of Information and Computer Sciences, University of Massachusetts, Amherst, MA, USA. hong.yu@umassmed.edu. 6. Department of Computer Science, University of Massachusetts, 220 Pawtucket St., Lowell, MA, 01854-2874, USA. hong.yu@umassmed.edu. 7. Department of Medicine, University of Massachusetts Medical School, Worcester, MA, USA. hong.yu@umassmed.edu. 8. Bedford VAMC, Bedford, MA, USA. hong.yu@umassmed.edu.
Abstract
INTRODUCTION: This work describes the Medication and Adverse Drug Events from Electronic Health Records (MADE 1.0) corpus and provides an overview of the MADE 1.0 2018 challenge for extracting medication, indication, and adverse drug events (ADEs) from electronic health record (EHR) notes. OBJECTIVE: The goal of MADE is to provide a set of common evaluation tasks to assess the state of the art for natural language processing (NLP) systems applied to EHRs supporting drug safety surveillance and pharmacovigilance. We also provide benchmarks on the MADE dataset using the system submissions received in the MADE 2018 challenge. METHODS: The MADE 1.0 challenge has released an expert-annotated cohort of medication and ADE information comprising 1089 fully de-identified longitudinal EHR notes from 21 randomly selected patients with cancer at the University of Massachusetts Memorial Hospital. Using this cohort as a benchmark, the MADE 1.0 challenge designed three shared NLP tasks. The named entity recognition (NER) task identifies medications and their attributes (dosage, route, duration, and frequency), indications, ADEs, and severity. The relation identification (RI) task identifies relations between the named entities: medication-indication, medication-ADE, and attribute relations. The third shared task (NER-RI) evaluates NLP models that perform the NER and RI tasks jointly. In total, 11 teams from four countries participated in at least one of the three shared tasks, and 41 system submissions were received in total. RESULTS: The best systems F1 scores for NER, RI, and NER-RI were 0.82, 0.86, and 0.61, respectively. Ensemble classifiers using the team submissions improved the performance further, with an F1 score of 0.85, 0.87, and 0.66 for the three tasks, respectively. CONCLUSION: MADE results show that recent progress in NLP has led to remarkable improvements in NER and RI tasks for the clinical domain. However, some room for improvement remains, particularly in the NER-RI task.
INTRODUCTION: This work describes the Medication and Adverse Drug Events from Electronic Health Records (MADE 1.0) corpus and provides an overview of the MADE 1.0 2018 challenge for extracting medication, indication, and adverse drug events (ADEs) from electronic health record (EHR) notes. OBJECTIVE: The goal of MADE is to provide a set of common evaluation tasks to assess the state of the art for natural language processing (NLP) systems applied to EHRs supporting drug safety surveillance and pharmacovigilance. We also provide benchmarks on the MADE dataset using the system submissions received in the MADE 2018 challenge. METHODS: The MADE 1.0 challenge has released an expert-annotated cohort of medication and ADE information comprising 1089 fully de-identified longitudinal EHR notes from 21 randomly selected patients with cancer at the University of Massachusetts Memorial Hospital. Using this cohort as a benchmark, the MADE 1.0 challenge designed three shared NLP tasks. The named entity recognition (NER) task identifies medications and their attributes (dosage, route, duration, and frequency), indications, ADEs, and severity. The relation identification (RI) task identifies relations between the named entities: medication-indication, medication-ADE, and attribute relations. The third shared task (NER-RI) evaluates NLP models that perform the NER and RI tasks jointly. In total, 11 teams from four countries participated in at least one of the three shared tasks, and 41 system submissions were received in total. RESULTS: The best systems F1 scores for NER, RI, and NER-RI were 0.82, 0.86, and 0.61, respectively. Ensemble classifiers using the team submissions improved the performance further, with an F1 score of 0.85, 0.87, and 0.66 for the three tasks, respectively. CONCLUSION: MADE results show that recent progress in NLP has led to remarkable improvements in NER and RI tasks for the clinical domain. However, some room for improvement remains, particularly in the NER-RI task.
Authors: Jerry H Gurwitz; Terry S Field; Leslie R Harrold; Jeffrey Rothschild; Kristin Debellis; Andrew C Seger; Cynthia Cadoret; Leslie S Fish; Lawrence Garber; Michael Kelleher; David W Bates Journal: JAMA Date: 2003-03-05 Impact factor: 56.272
Authors: D W Bates; D J Cullen; N Laird; L A Petersen; S D Small; D Servi; G Laffel; B J Sweitzer; B F Shea; R Hallisey Journal: JAMA Date: 1995-07-05 Impact factor: 56.272
Authors: Qi Li; Kristin Melton; Todd Lingren; Eric S Kirkendall; Eric Hall; Haijun Zhai; Yizhao Ni; Megan Kaiser; Laura Stoutenborough; Imre Solti Journal: J Am Med Inform Assoc Date: 2014-01-08 Impact factor: 4.497
Authors: Andrew Nashed; Shijun Zhang; Chien-Wei Chiang; M Zitu; Gregory A Otterson; Carolyn J Presley; Kari Kendra; Sandip H Patel; Andrew Johns; Mingjia Li; Madison Grogan; Gabrielle Lopez; Dwight H Owen; Lang Li Journal: Cancer Immunol Immunother Date: 2021-02-24 Impact factor: 6.968