Benjamin D Wissel1, Hansel M Greiner2,3, Tracy A Glauser2,3, Katherine D Holland-Bouley2,3, Francesco T Mangano2,4, Daniel Santel1, Robert Faist1, Nanhua Zhang2,5, John P Pestian1,2, Rhonda D Szczesniak2,5, Judith W Dexheimer1,2,6. 1. Department of Biomedical Informatics, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio. 2. Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio. 3. Division of Neurology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio. 4. Division of Neurosurgery, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio. 5. Division of Biostatistics & Epidemiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio. 6. Division of Emergency Medicine, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio.
Abstract
OBJECTIVE: Delay to resective epilepsy surgery results in avoidable disease burden and increased risk of mortality. The objective was to prospectively validate a natural language processing (NLP) application that uses provider notes to assign epilepsy surgery candidacy scores. METHODS: The application was trained on notes from (1) patients with a diagnosis of epilepsy and a history of resective epilepsy surgery and (2) patients who were seizure-free without surgery. The testing set included all patients with unknown surgical candidacy status and an upcoming neurology visit. Training and testing sets were updated weekly for 1 year. One- to three-word phrases contained in patients' notes were used as features. Patients prospectively identified by the application as candidates for surgery were manually reviewed by two epileptologists. Performance metrics were defined by comparing NLP-derived surgical candidacy scores with surgical candidacy status from expert chart review. RESULTS: The training set was updated weekly and included notes from a mean of 519 ± 67 patients. The area under the receiver operating characteristic curve (AUC) from 10-fold cross-validation was 0.90 ± 0.04 (range = 0.83-0.96) and improved by 0.002 per week (P < .001) as new patients were added to the training set. Of the 6395 patients who visited the neurology clinic, 4211 (67%) were evaluated by the model. The prospective AUC on this test set was 0.79 (95% confidence interval [CI] = 0.62-0.96). Using the optimal surgical candidacy score threshold, sensitivity was 0.80 (95% CI = 0.29-0.99), specificity was 0.77 (95% CI = 0.64-0.88), positive predictive value was 0.25 (95% CI = 0.07-0.52), and negative predictive value was 0.98 (95% CI = 0.87-1.00). The number needed to screen was 5.6. SIGNIFICANCE: An electronic health record-integrated NLP application can accurately assign surgical candidacy scores to patients in a clinical setting. Wiley Periodicals, Inc.
OBJECTIVE: Delay to resective epilepsy surgery results in avoidable disease burden and increased risk of mortality. The objective was to prospectively validate a natural language processing (NLP) application that uses provider notes to assign epilepsy surgery candidacy scores. METHODS: The application was trained on notes from (1) patients with a diagnosis of epilepsy and a history of resective epilepsy surgery and (2) patients who were seizure-free without surgery. The testing set included all patients with unknown surgical candidacy status and an upcoming neurology visit. Training and testing sets were updated weekly for 1 year. One- to three-word phrases contained in patients' notes were used as features. Patients prospectively identified by the application as candidates for surgery were manually reviewed by two epileptologists. Performance metrics were defined by comparing NLP-derived surgical candidacy scores with surgical candidacy status from expert chart review. RESULTS: The training set was updated weekly and included notes from a mean of 519 ± 67 patients. The area under the receiver operating characteristic curve (AUC) from 10-fold cross-validation was 0.90 ± 0.04 (range = 0.83-0.96) and improved by 0.002 per week (P < .001) as new patients were added to the training set. Of the 6395 patients who visited the neurology clinic, 4211 (67%) were evaluated by the model. The prospective AUC on this test set was 0.79 (95% confidence interval [CI] = 0.62-0.96). Using the optimal surgical candidacy score threshold, sensitivity was 0.80 (95% CI = 0.29-0.99), specificity was 0.77 (95% CI = 0.64-0.88), positive predictive value was 0.25 (95% CI = 0.07-0.52), and negative predictive value was 0.98 (95% CI = 0.87-1.00). The number needed to screen was 5.6. SIGNIFICANCE: An electronic health record-integrated NLP application can accurately assign surgical candidacy scores to patients in a clinical setting. Wiley Periodicals, Inc.
Authors: Chiara Fois; Stjepana Kovac; Aytakin Khalil; Gülnur Tekgöl Uzuner; Beate Diehl; Tim Wehner; John S Duncan; Matthew C Walker Journal: J Neurol Neurosurg Psychiatry Date: 2015-05-02 Impact factor: 10.154
Authors: Benjamin D Wissel; Hansel M Greiner; Tracy A Glauser; Francesco T Mangano; Daniel Santel; John P Pestian; Rhonda D Szczesniak; Judith W Dexheimer Journal: Epilepsia Date: 2019-08-23 Impact factor: 5.864
Authors: J Engel; S Wiebe; J French; M Sperling; P Williamson; D Spencer; R Gumnit; C Zahn; E Westbrook; B Enos Journal: Neurology Date: 2003-02-25 Impact factor: 9.910
Authors: A T Berg; J Langfitt; S Shinnar; B G Vickrey; M R Sperling; T Walczak; C Bazil; S V Pacia; S S Spencer Journal: Neurology Date: 2003-01-28 Impact factor: 9.910
Authors: Karel G M Moons; Douglas G Altman; Johannes B Reitsma; John P A Ioannidis; Petra Macaskill; Ewout W Steyerberg; Andrew J Vickers; David F Ransohoff; Gary S Collins Journal: Ann Intern Med Date: 2015-01-06 Impact factor: 25.391
Authors: Benjamin D Wissel; Hansel M Greiner; Tracy A Glauser; John P Pestian; Andrew J Kemme; Daniel Santel; David M Ficker; Francesco T Mangano; Rhonda D Szczesniak; Judith W Dexheimer Journal: Acta Neurol Scand Date: 2021-03-26 Impact factor: 3.915