Hans Moen1, Laura-Maria Peltonen2, Juho Heimonen3, Antti Airola4, Tapio Pahikkala5, Tapio Salakoski6, Sanna Salanterä7. 1. Department of Computer and Information Science, Norwegian University of Science and Technology, Sem Saelands vei 9, 7491 Trondheim, Norway; Department of Information Technology, University of Turku, Joukahaisenkatu 3-5, 20520 Turku, Finland; Department of Nursing Science, University of Turku, Lemminkäisenkatu 1, 20520 Turku, Finland. Electronic address: hans.moen@idi.ntnu.no. 2. Department of Nursing Science, University of Turku, Lemminkäisenkatu 1, 20520 Turku, Finland; Turku University Hospital, Kiinamyllynkatu 4-8, 20521 Turku, Finland. Electronic address: lmemur@utu.fi. 3. Department of Information Technology, University of Turku, Joukahaisenkatu 3-5, 20520 Turku, Finland; Turku Centre for Computer Science (TUCS), Joukahaisenkatu 3-5, 20520 Turku, Finland. Electronic address: juaheim@utu.fi. 4. Department of Information Technology, University of Turku, Joukahaisenkatu 3-5, 20520 Turku, Finland. Electronic address: ajairo@utu.fi. 5. Department of Information Technology, University of Turku, Joukahaisenkatu 3-5, 20520 Turku, Finland; Turku Centre for Computer Science (TUCS), Joukahaisenkatu 3-5, 20520 Turku, Finland. Electronic address: aatapa@utu.fi. 6. Department of Information Technology, University of Turku, Joukahaisenkatu 3-5, 20520 Turku, Finland; Turku Centre for Computer Science (TUCS), Joukahaisenkatu 3-5, 20520 Turku, Finland. Electronic address: tapio.salakoski@utu.fi. 7. Department of Nursing Science, University of Turku, Lemminkäisenkatu 1, 20520 Turku, Finland; Turku University Hospital, Kiinamyllynkatu 4-8, 20521 Turku, Finland. Electronic address: sansala@utu.fi.
Abstract
OBJECTIVE: A major source of information available in electronic health record (EHR) systems are the clinical free text notes documenting patient care. Managing this information is time-consuming for clinicians. Automatic text summarisation could assist clinicians in obtaining an overview of the free text information in ongoing care episodes, as well as in writing final discharge summaries. We present a study of automated text summarisation of clinical notes. It looks to identify which methods are best suited for this task and whether it is possible to automatically evaluate the quality differences of summaries produced by different methods in an efficient and reliable way. METHODS AND MATERIALS: The study is based on material consisting of 66,884 care episodes from EHRs of heart patients admitted to a university hospital in Finland between 2005 and 2009. We present novel extractive text summarisation methods for summarising the free text content of care episodes. Most of these methods rely on word space models constructed using distributional semantic modelling. The summarisation effectiveness is evaluated using an experimental automatic evaluation approach incorporating well-known ROUGE measures. We also developed a manual evaluation scheme to perform a meta-evaluation on the ROUGE measures to see if they reflect the opinions of health care professionals. RESULTS: The agreement between the human evaluators is good (ICC=0.74, p<0.001), demonstrating the stability of the proposed manual evaluation method. Furthermore, the correlation between the manual and automated evaluations are high (> 0.90 Spearman's rho). Three of the presented summarisation methods ('Composite', 'Case-Based' and 'Translate') significantly outperform the other methods for all ROUGE measures (p<0.05, Wilcoxon signed-rank test and Bonferroni correction). CONCLUSION: The results indicate the feasibility of the automated summarisation of care episodes. Moreover, the high correlation between manual and automated evaluations suggests that the less labour-intensive automated evaluations can be used as a proxy for human evaluations when developing summarisation methods. This is of significant practical value for summarisation method development, because manual evaluation cannot be afforded for every variation of the summarisation methods. Instead, one can resort to automatic evaluation during the method development process.
OBJECTIVE: A major source of information available in electronic health record (EHR) systems are the clinical free text notes documenting patient care. Managing this information is time-consuming for clinicians. Automatic text summarisation could assist clinicians in obtaining an overview of the free text information in ongoing care episodes, as well as in writing final discharge summaries. We present a study of automated text summarisation of clinical notes. It looks to identify which methods are best suited for this task and whether it is possible to automatically evaluate the quality differences of summaries produced by different methods in an efficient and reliable way. METHODS AND MATERIALS: The study is based on material consisting of 66,884 care episodes from EHRs of heart patients admitted to a university hospital in Finland between 2005 and 2009. We present novel extractive text summarisation methods for summarising the free text content of care episodes. Most of these methods rely on word space models constructed using distributional semantic modelling. The summarisation effectiveness is evaluated using an experimental automatic evaluation approach incorporating well-known ROUGE measures. We also developed a manual evaluation scheme to perform a meta-evaluation on the ROUGE measures to see if they reflect the opinions of health care professionals. RESULTS: The agreement between the human evaluators is good (ICC=0.74, p<0.001), demonstrating the stability of the proposed manual evaluation method. Furthermore, the correlation between the manual and automated evaluations are high (> 0.90 Spearman's rho). Three of the presented summarisation methods ('Composite', 'Case-Based' and 'Translate') significantly outperform the other methods for all ROUGE measures (p<0.05, Wilcoxon signed-rank test and Bonferroni correction). CONCLUSION: The results indicate the feasibility of the automated summarisation of care episodes. Moreover, the high correlation between manual and automated evaluations suggests that the less labour-intensive automated evaluations can be used as a proxy for human evaluations when developing summarisation methods. This is of significant practical value for summarisation method development, because manual evaluation cannot be afforded for every variation of the summarisation methods. Instead, one can resort to automatic evaluation during the method development process.
Keywords:
Automatic text summarisation; Clinical text processing; Distributional semantics; Electronic health records; Summarisation evaluation; Word space models
Authors: Hind Bitar; Amal Babour; Fatema Nafa; Ohoud Alzamzami; Sarah Alismail Journal: Int J Environ Res Public Health Date: 2022-07-01 Impact factor: 4.614
Authors: Honghan Wu; Anika Oellrich; Christine Girges; Bernard de Bono; Tim J P Hubbard; Richard J B Dobson Journal: Database (Oxford) Date: 2017-01-01 Impact factor: 3.451