Carlton R Moore1, Ashraf Farrag, Evan Ashkin. 1. From the *Division of General Medicine and Clinical Epidemiology, Department of Medicine, School of Medicine, †The North Carolina Translational and Clinical Sciences Center, and ‡Department of family Medicine, School of Medicine, University of North Carolina, Chapel Hill, North Carolina.
Abstract
OBJECTIVES: Numerous studies show that follow-up of abnormal cancer screening results, such as mammography and Papanicolaou (Pap) smears, is frequently not performed in a timely manner. A contributing factor is that abnormal results may go unrecognized because they are buried in free-text documents in electronic medical records (EMRs), and, as a result, patients are lost to follow-up. By identifying abnormal results from free-text reports in EMRs and generating alerts to clinicians, natural language processing (NLP) technology has the potential for improving patient care. The goal of the current study was to evaluate the performance of NLP software for extracting abnormal results from free-text mammography and Pap smear reports stored in an EMR. METHODS: A sample of 421 and 500 free-text mammography and Pap reports, respectively, were manually reviewed by a physician, and the results were categorized for each report. We tested the performance of NLP to extract results from the reports. The 2 assessments (criterion standard versus NLP) were compared to determine the precision, recall, and accuracy of NLP. RESULTS: When NLP was compared with manual review for mammography reports, the results were as follows: precision, 98% (96%-99%); recall, 100% (98%-100%); and accuracy, 98% (96%-99%). For Pap smear reports, the precision, recall, and accuracy of NLP were all 100%. CONCLUSIONS: Our study developed NLP models that accurately extract abnormal results from mammography and Pap smear reports. Plans include using NLP technology to generate real-time alerts and reminders for providers to facilitate timely follow-up of abnormal results.
OBJECTIVES: Numerous studies show that follow-up of abnormal cancer screening results, such as mammography and Papanicolaou (Pap) smears, is frequently not performed in a timely manner. A contributing factor is that abnormal results may go unrecognized because they are buried in free-text documents in electronic medical records (EMRs), and, as a result, patients are lost to follow-up. By identifying abnormal results from free-text reports in EMRs and generating alerts to clinicians, natural language processing (NLP) technology has the potential for improving patient care. The goal of the current study was to evaluate the performance of NLP software for extracting abnormal results from free-text mammography and Pap smear reports stored in an EMR. METHODS: A sample of 421 and 500 free-text mammography and Pap reports, respectively, were manually reviewed by a physician, and the results were categorized for each report. We tested the performance of NLP to extract results from the reports. The 2 assessments (criterion standard versus NLP) were compared to determine the precision, recall, and accuracy of NLP. RESULTS: When NLP was compared with manual review for mammography reports, the results were as follows: precision, 98% (96%-99%); recall, 100% (98%-100%); and accuracy, 98% (96%-99%). For Pap smear reports, the precision, recall, and accuracy of NLP were all 100%. CONCLUSIONS: Our study developed NLP models that accurately extract abnormal results from mammography and Pap smear reports. Plans include using NLP technology to generate real-time alerts and reminders for providers to facilitate timely follow-up of abnormal results.
Authors: Beth A Jones; Amy Dailey; Lisa Calvocoressi; Kam Reams; Stanislav V Kasl; Carol Lee; Helen Hsu Journal: Cancer Causes Control Date: 2005-09 Impact factor: 2.506
Authors: Annette J Johnson; Michael Y M Chen; J Shannon Swan; Kimberly E Applegate; Benjamin Littenberg Journal: Radiology Date: 2009-08-25 Impact factor: 11.105
Authors: Margaret L Plews-Ogan; Mohan M Nadkarni; Sue Forren; Darlene Leon; Donna White; Don Marineau; John B Schorling; Joel M Schectman Journal: J Gen Intern Med Date: 2004-07 Impact factor: 5.128
Authors: Archana Laxmisan; Dean F Sittig; Kenneth Pietz; Donna Espadas; Bhuvaneswari Krishnan; Hardeep Singh Journal: Med Care Date: 2012-10 Impact factor: 2.983