PURPOSE: The objective was to develop a natural language processing (NLP) algorithm to identify vaccine-related anaphylaxis from plain-text clinical notes, and to implement the algorithm at five health care systems in the Vaccine Safety Datalink. METHODS: The NLP algorithm was developed using an internal NLP tool and training dataset of 311 potential anaphylaxis cases from Kaiser Permanente Southern California (KPSC). We applied the algorithm to the notes of another 731 potential cases (423 from KPSC; 308 from other sites) with relevant codes (ICD-9-CM diagnosis codes for anaphylaxis, vaccine adverse reactions, and allergic reactions; Healthcare Common Procedure Coding System codes for epinephrine administration). NLP results were compared against a reference standard of chart reviewed and adjudicated cases. The algorithm was then separately applied to the notes of 6 427 359 KPSC vaccination visits (9 402 194 vaccine doses) without relevant codes. RESULTS: At KPSC, NLP identified 12 of 16 true vaccine-related cases and achieved a sensitivity of 75.0%, specificity of 98.5%, positive predictive value (PPV) of 66.7%, and negative predictive value of 99.0% when applied to notes of patients with relevant diagnosis codes. NLP did not identify the five true cases at other sites. When NLP was applied to the notes of KPSC patients without relevant codes, it captured eight additional true cases confirmed by chart review and adjudication. CONCLUSIONS: The current study demonstrated the potential to apply rule-based NLP algorithms to clinical notes to identify anaphylaxis cases. Increasing the size of training data, including clinical notes from all participating study sites in the training data, and preprocessing the clinical notes to handle special characters could improve the performance of the NLP algorithms. We recommend adding an NLP process followed by manual chart review in future vaccine safety studies to improve sensitivity and efficiency.
PURPOSE: The objective was to develop a natural language processing (NLP) algorithm to identify vaccine-related anaphylaxis from plain-text clinical notes, and to implement the algorithm at five health care systems in the Vaccine Safety Datalink. METHODS: The NLP algorithm was developed using an internal NLP tool and training dataset of 311 potential anaphylaxis cases from Kaiser Permanente Southern California (KPSC). We applied the algorithm to the notes of another 731 potential cases (423 from KPSC; 308 from other sites) with relevant codes (ICD-9-CM diagnosis codes for anaphylaxis, vaccine adverse reactions, and allergic reactions; Healthcare Common Procedure Coding System codes for epinephrine administration). NLP results were compared against a reference standard of chart reviewed and adjudicated cases. The algorithm was then separately applied to the notes of 6 427 359 KPSC vaccination visits (9 402 194 vaccine doses) without relevant codes. RESULTS: At KPSC, NLP identified 12 of 16 true vaccine-related cases and achieved a sensitivity of 75.0%, specificity of 98.5%, positive predictive value (PPV) of 66.7%, and negative predictive value of 99.0% when applied to notes of patients with relevant diagnosis codes. NLP did not identify the five true cases at other sites. When NLP was applied to the notes of KPSC patients without relevant codes, it captured eight additional true cases confirmed by chart review and adjudication. CONCLUSIONS: The current study demonstrated the potential to apply rule-based NLP algorithms to clinical notes to identify anaphylaxis cases. Increasing the size of training data, including clinical notes from all participating study sites in the training data, and preprocessing the clinical notes to handle special characters could improve the performance of the NLP algorithms. We recommend adding an NLP process followed by manual chart review in future vaccine safety studies to improve sensitivity and efficiency.
Authors: Harvey J Murff; Alan J Forster; Josh F Peterson; Julie M Fiskio; Heather L Heiman; David W Bates Journal: J Am Med Inform Assoc Date: 2003-03-28 Impact factor: 4.497
Authors: Taxiarchis Botsis; Michael D Nguyen; Emily Jane Woo; Marianthi Markatou; Robert Ball Journal: J Am Med Inform Assoc Date: 2011-06-27 Impact factor: 4.497
Authors: Robert Ball; Sengwee Toh; Jamie Nolan; Kevin Haynes; Richard Forshee; Taxiarchis Botsis Journal: Pharmacoepidemiol Drug Saf Date: 2018-08-28 Impact factor: 2.890
Authors: Chengyi Zheng; Wei Yu; Fagen Xie; Wansu Chen; Cheryl Mercado; Lina S Sy; Lei Qian; Sungching Glenn; Gina Lee; Hung Fu Tseng; Jonathan Duffy; Lisa A Jackson; Matthew F Daley; Brad Crane; Huong Q McLean; Steven J Jacobsen Journal: Int J Med Inform Date: 2019-04-13 Impact factor: 4.046
Authors: James Baggs; Julianne Gee; Edwin Lewis; Gabrielle Fowler; Patti Benson; Tracy Lieu; Allison Naleway; Nicola P Klein; Roger Baxter; Edward Belongia; Jason Glanz; Simon J Hambidge; Steven J Jacobsen; Lisa Jackson; Jim Nordin; Eric Weintraub Journal: Pediatrics Date: 2011-04-18 Impact factor: 7.124
Authors: Michael M McNeil; Eric S Weintraub; Jonathan Duffy; Lakshmi Sukumaran; Steven J Jacobsen; Nicola P Klein; Simon J Hambidge; Grace M Lee; Lisa A Jackson; Stephanie A Irving; Jennifer P King; Elyse O Kharbanda; Robert A Bednarczyk; Frank DeStefano Journal: J Allergy Clin Immunol Date: 2015-10-06 Impact factor: 10.793
Authors: Chengyi Zheng; Jonathan Duffy; In-Lu Amy Liu; Lina S Sy; Wansu Chen; Lei Qian; Ronald A Navarro; Denison S Ryan; Sunhea S Kim; Cheryl Mercado; Steven J Jacobsen Journal: Ann Intern Med Date: 2022-03-22 Impact factor: 25.391
Authors: Chengyi Zheng; Jonathan Duffy; In-Lu Amy Liu; Lina S Sy; Ronald A Navarro; Sunhea S Kim; Denison S Ryan; Wansu Chen; Lei Qian; Cheryl Mercado; Steven J Jacobsen Journal: JMIR Public Health Surveill Date: 2022-05-24
Authors: Fagen Xie; Nehaa Khadka; Michael J Fassett; Vicki Y Chiu; Chantal C Avila; Jiaxiao Shi; Meiyu Yeh; Aniket Kawatkar; Nana A Mensah; David A Sacks; Darios Getahun Journal: JMIR Med Inform Date: 2022-09-06
Authors: Carlos R Oliveira; Patrick Niccolai; Anette Michelle Ortiz; Sangini S Sheth; Eugene D Shapiro; Linda M Niccolai; Cynthia A Brandt Journal: JMIR Med Inform Date: 2020-11-03