OBJECTIVES: This study evaluates the accuracy and portability of a natural language processing (NLP) tool for extracting clinical findings of influenza from clinical notes across two large healthcare systems. Effectiveness is evaluated on how well NLP supports downstream influenza case-detection for disease surveillance. METHODS: We independently developed two NLP parsers, one at Intermountain Healthcare (IH) in Utah and the other at University of Pittsburgh Medical Center (UPMC) using local clinical notes from emergency department (ED) encounters of influenza. We measured NLP parser performance for the presence and absence of 70 clinical findings indicative of influenza. We then developed Bayesian network models from NLP processed reports and tested their ability to discriminate among cases of (1) influenza, (2) non-influenza influenza-like illness (NI-ILI), and (3) 'other' diagnosis. RESULTS: On Intermountain Healthcare reports, recall and precision of the IH NLP parser were 0.71 and 0.75, respectively, and UPMC NLP parser, 0.67 and 0.79. On University of Pittsburgh Medical Center reports, recall and precision of the UPMC NLP parser were 0.73 and 0.80, respectively, and IH NLP parser, 0.53 and 0.80. Bayesian case-detection performance measured by AUROC for influenza versus non-influenza on Intermountain Healthcare cases was 0.93 (using IH NLP parser) and 0.93 (using UPMC NLP parser). Case-detection on University of Pittsburgh Medical Center cases was 0.95 (using UPMC NLP parser) and 0.83 (using IH NLP parser). For influenza versus NI-ILI on Intermountain Healthcare cases performance was 0.70 (using IH NLP parser) and 0.76 (using UPMC NLP parser). On University of Pisstburgh Medical Center cases, 0.76 (using UPMC NLP parser) and 0.65 (using IH NLP parser). CONCLUSION: In all but one instance (influenza versus NI-ILI using IH cases), local parsers were more effective at supporting case-detection although performances of non-local parsers were reasonable.
OBJECTIVES: This study evaluates the accuracy and portability of a natural language processing (NLP) tool for extracting clinical findings of influenza from clinical notes across two large healthcare systems. Effectiveness is evaluated on how well NLP supports downstream influenza case-detection for disease surveillance. METHODS: We independently developed two NLP parsers, one at Intermountain Healthcare (IH) in Utah and the other at University of Pittsburgh Medical Center (UPMC) using local clinical notes from emergency department (ED) encounters of influenza. We measured NLP parser performance for the presence and absence of 70 clinical findings indicative of influenza. We then developed Bayesian network models from NLP processed reports and tested their ability to discriminate among cases of (1) influenza, (2) non-influenza influenza-like illness (NI-ILI), and (3) 'other' diagnosis. RESULTS: On Intermountain Healthcare reports, recall and precision of the IH NLP parser were 0.71 and 0.75, respectively, and UPMC NLP parser, 0.67 and 0.79. On University of Pittsburgh Medical Center reports, recall and precision of the UPMC NLP parser were 0.73 and 0.80, respectively, and IH NLP parser, 0.53 and 0.80. Bayesian case-detection performance measured by AUROC for influenza versus non-influenza on Intermountain Healthcare cases was 0.93 (using IH NLP parser) and 0.93 (using UPMC NLP parser). Case-detection on University of Pittsburgh Medical Center cases was 0.95 (using UPMC NLP parser) and 0.83 (using IH NLP parser). For influenza versus NI-ILI on Intermountain Healthcare cases performance was 0.70 (using IH NLP parser) and 0.76 (using UPMC NLP parser). On University of Pisstburgh Medical Center cases, 0.76 (using UPMC NLP parser) and 0.65 (using IH NLP parser). CONCLUSION: In all but one instance (influenza versus NI-ILI using IH cases), local parsers were more effective at supporting case-detection although performances of non-local parsers were reasonable.
Keywords:
Natural language processing; case detection; disease surveillance; generalizability; portability
Authors: Per H Gesteland; Michael M Wagner; Wendy W Chapman; Jeremy U Espino; Fu-Chiang Tsui; Reed M Gardner; Robert T Rolfs; Virginia Dato; Brent C James; Peter J Haug Journal: Proc AMIA Symp Date: 2002
Authors: Joseph Lombardo; Howard Burkom; Eugene Elbert; Steven Magruder; Sheryl Happel Lewis; Wayne Loschen; James Sari; Carol Sniegoski; Richard Wojcik; Julie Pavlin Journal: J Urban Health Date: 2003-06 Impact factor: 3.671
Authors: Joshua C Denny; Anderson Spickard; Kevin B Johnson; Neeraja B Peterson; Josh F Peterson; Randolph A Miller Journal: J Am Med Inform Assoc Date: 2009-08-28 Impact factor: 4.497
Authors: Wendy W Chapman; Prakash M Nadkarni; Lynette Hirschman; Leonard W D'Avolio; Guergana K Savova; Ozlem Uzuner Journal: J Am Med Inform Assoc Date: 2011 Sep-Oct Impact factor: 4.497
Authors: Nathan C Dean; Barbara E Jones; Jeffrey P Ferraro; Caroline G Vines; Peter J Haug Journal: JAMA Intern Med Date: 2013-04-22 Impact factor: 21.873
Authors: Jeffrey P Ferraro; Hal Daumé; Scott L Duvall; Wendy W Chapman; Henk Harkema; Peter J Haug Journal: J Am Med Inform Assoc Date: 2013-03-13 Impact factor: 4.497
Authors: Victor M Castro; Jessica Minnier; Shawn N Murphy; Isaac Kohane; Susanne E Churchill; Vivian Gainer; Tianxi Cai; Alison G Hoffnagle; Yael Dai; Stefanie Block; Sydney R Weill; Mireya Nadal-Vicens; Alisha R Pollastri; J Niels Rosenquist; Sergey Goryachev; Dost Ongur; Pamela Sklar; Roy H Perlis; Jordan W Smoller Journal: Am J Psychiatry Date: 2014-12-12 Impact factor: 18.112
Authors: Patrick C Burke; Rachel Benish Shirley; Jacob Raciniewski; James F Simon; Robert Wyllie; Thomas G Fraser Journal: Appl Clin Inform Date: 2020-08-26 Impact factor: 2.342
Authors: Steven Horng; Nathaniel R Greenbaum; Larry A Nathanson; James C McClay; Foster R Goss; Jeffrey A Nielson Journal: Appl Clin Inform Date: 2019-06-12 Impact factor: 2.342
Authors: Daniel J Feller; Oliver J Bear Don't Walk Iv; Jason Zucker; Michael T Yin; Peter Gordon; Noémie Elhadad Journal: Appl Clin Inform Date: 2020-03-04 Impact factor: 2.342