Andrew Yu Li1, Nikki Elliot2. 1. Department of Radiology, Canterbury District Health Board, Christchurch, New Zealand. 2. The Canterbury Initiative, Canterbury District Health Board, Christchurch, New Zealand.
Abstract
INTRODUCTION: Natural language processing (NLP) is an emerging tool which has the ability to automate data extraction from large volumes of unstructured text. One of the main described uses of NLP in radiology is cohort building for epidemiological studies. This study aims to assess the accuracy of NLP in identifying a group of patients positive for ureteric stones on Computed Tomography - Kidneys, Ureter, Bladder (CT KUB) reports. METHODS: Retrospective review of all CT KUB reports in a single calendar year. A locally available NLP tool was used to automatically classify the reports based on positivity for ureteric stones. This was validated by manual review and refined to maximize the accuracy of stone detection. RESULTS: A total of 1874 CT KUB reports were identified. Manual classification of ureteric stone positivity was 36% compared with 27% using NLP. The accuracy of NLP was 85% with a sensitivity of 66% and specificity of 95%. Incorrect classification was due to misspellings, variable syntax, terminology, pluralization and the inability to exclude clinical request details from the search algorithm. CONCLUSIONS: Our NLP tool demonstrated high specificity but low sensitivity at identifying CT KUB reports that are positive for ureteric stones. This was attributable to the lack of feature extraction tools tailored for analysing radiology text, incompleteness of the medical lexicon database and heterogeneity of unstructured reports. Improvements in these areas will help improve data extraction accuracy.
INTRODUCTION: Natural language processing (NLP) is an emerging tool which has the ability to automate data extraction from large volumes of unstructured text. One of the main described uses of NLP in radiology is cohort building for epidemiological studies. This study aims to assess the accuracy of NLP in identifying a group of patients positive for ureteric stones on Computed Tomography - Kidneys, Ureter, Bladder (CT KUB) reports. METHODS: Retrospective review of all CT KUB reports in a single calendar year. A locally available NLP tool was used to automatically classify the reports based on positivity for ureteric stones. This was validated by manual review and refined to maximize the accuracy of stone detection. RESULTS: A total of 1874 CT KUB reports were identified. Manual classification of ureteric stone positivity was 36% compared with 27% using NLP. The accuracy of NLP was 85% with a sensitivity of 66% and specificity of 95%. Incorrect classification was due to misspellings, variable syntax, terminology, pluralization and the inability to exclude clinical request details from the search algorithm. CONCLUSIONS: Our NLP tool demonstrated high specificity but low sensitivity at identifying CT KUB reports that are positive for ureteric stones. This was attributable to the lack of feature extraction tools tailored for analysing radiology text, incompleteness of the medical lexicon database and heterogeneity of unstructured reports. Improvements in these areas will help improve data extraction accuracy.
Authors: Arlene Casey; Emma Davidson; Michael Poon; Hang Dong; Daniel Duma; Andreas Grivas; Claire Grover; Víctor Suárez-Paniagua; Richard Tobin; William Whiteley; Honghan Wu; Beatrice Alex Journal: BMC Med Inform Decis Mak Date: 2021-06-03 Impact factor: 2.796