Sam Henry1, Kevin Buchan2, Michele Filannino1,3, Amber Stubbs4, Ozlem Uzuner1,3,5. 1. Department is Information Sciences and Technology, George Mason University, Fairfax, Virginia, USA. 2. Department of Information Science, University at Albany - State University of New York, Albany, New York, USA. 3. Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA. 4. Department is Mathematics and Computer Science, Simmons University, Boston, Massachusetts, USA. 5. Department of Biomedical Informatics, Harvard Medical School, Boston, Massachusetts, USA.
Abstract
OBJECTIVE: This article summarizes the preparation, organization, evaluation, and results of Track 2 of the 2018 National NLP Clinical Challenges shared task. Track 2 focused on extraction of adverse drug events (ADEs) from clinical records and evaluated 3 tasks: concept extraction, relation classification, and end-to-end systems. We perform an analysis of the results to identify the state of the art in these tasks, learn from it, and build on it. MATERIALS AND METHODS: For all tasks, teams were given raw text of narrative discharge summaries, and in all the tasks, participants proposed deep learning-based methods with hand-designed features. In the concept extraction task, participants used sequence labelling models (bidirectional long short-term memory being the most popular), whereas in the relation classification task, they also experimented with instance-based classifiers (namely support vector machines and rules). Ensemble methods were also popular. RESULTS: A total of 28 teams participated in task 1, with 21 teams in tasks 2 and 3. The best performing systems set a high performance bar with F1 scores of 0.9418 for concept extraction, 0.9630 for relation classification, and 0.8905 for end-to-end. However, the results were much lower for concepts and relations of Reasons and ADEs. These were often missed because local context is insufficient to identify them. CONCLUSIONS: This challenge shows that clinical concept extraction and relation classification systems have a high performance for many concept types, but significant improvement is still required for ADEs and Reasons. Incorporating the larger context or outside knowledge will likely improve the performance of future systems.
OBJECTIVE: This article summarizes the preparation, organization, evaluation, and results of Track 2 of the 2018 National NLP Clinical Challenges shared task. Track 2 focused on extraction of adverse drug events (ADEs) from clinical records and evaluated 3 tasks: concept extraction, relation classification, and end-to-end systems. We perform an analysis of the results to identify the state of the art in these tasks, learn from it, and build on it. MATERIALS AND METHODS: For all tasks, teams were given raw text of narrative discharge summaries, and in all the tasks, participants proposed deep learning-based methods with hand-designed features. In the concept extraction task, participants used sequence labelling models (bidirectional long short-term memory being the most popular), whereas in the relation classification task, they also experimented with instance-based classifiers (namely support vector machines and rules). Ensemble methods were also popular. RESULTS: A total of 28 teams participated in task 1, with 21 teams in tasks 2 and 3. The best performing systems set a high performance bar with F1 scores of 0.9418 for concept extraction, 0.9630 for relation classification, and 0.8905 for end-to-end. However, the results were much lower for concepts and relations of Reasons and ADEs. These were often missed because local context is insufficient to identify them. CONCLUSIONS: This challenge shows that clinical concept extraction and relation classification systems have a high performance for many concept types, but significant improvement is still required for ADEs and Reasons. Incorporating the larger context or outside knowledge will likely improve the performance of future systems.
Authors: Ozlem Uzuner; Andreea Bodnari; Shuying Shen; Tyler Forbush; John Pestian; Brett R South Journal: J Am Med Inform Assoc Date: 2012-02-24 Impact factor: 4.497
Authors: Hannah Eyre; Alec B Chapman; Kelly S Peterson; Jianlin Shi; Patrick R Alba; Makoto M Jones; Tamára L Box; Scott L DuVall; Olga V Patterson Journal: AMIA Annu Symp Proc Date: 2022-02-21
Authors: Shorabuddin Syed; Adam Jackson Angel; Hafsa Bareen Syeda; Carole Franc Jennings; Joseph VanScoy; Mahanazuddin Syed; Melody Greer; Sudeepa Bhattacharyya; Shaymaa Al-Shukri; Meredith Zozus; Fred Prior; Benjamin Tharian Journal: Biomed Eng Syst Technol Int Jt Conf BIOSTEC Revis Sel Pap Date: 2022-02
Authors: Jingcheng Du; Yang Xiang; Madhuri Sankaranarayanapillai; Meng Zhang; Jingqi Wang; Yuqi Si; Huy Anh Pham; Hua Xu; Yong Chen; Cui Tao Journal: J Am Med Inform Assoc Date: 2021-07-14 Impact factor: 4.497