Glenn T Gobbel1, Jennifer Garvin2, Ruth Reeves3, Robert M Cronin4, Julia Heavirland5, Jenifer Williams5, Allison Weaver5, Shrimalini Jayaramaraja6, Dario Giuse7, Theodore Speroff8, Steven H Brown3, Hua Xu9, Michael E Matheny10. 1. Department of Veterans Affairs Medical Center, Geriatric Research, Education and Clinical Center (GRECC), Nashville, Tennessee, USA Department of Biomedical Informatics, Vanderbilt University School of Medicine, Nashville, Tennessee, USA Division of General Internal Medicine & Public Health, Department of Medicine, Vanderbilt University School of Medicine, Nashville, Tennessee, USA. 2. IDEAS Center SLC VA Healthcare System, Salt Lake City, Utah, USA Division of Epidemiology, University of Utah School of Medicine, Salt Lake City, Utah, USA Department of Biomedical Informatics, University of Utah School of Medicine, Salt Lake City, Utah, USA Department of Veterans Affairs Medical Center, Geriatric Research, Education and Clinical Center (GRECC), Salt Lake City, Utah, USA. 3. Department of Veterans Affairs Medical Center, Geriatric Research, Education and Clinical Center (GRECC), Nashville, Tennessee, USA Department of Biomedical Informatics, Vanderbilt University School of Medicine, Nashville, Tennessee, USA. 4. Department of Biomedical Informatics, Vanderbilt University School of Medicine, Nashville, Tennessee, USA Division of General Internal Medicine & Public Health, Department of Medicine, Vanderbilt University School of Medicine, Nashville, Tennessee, USA. 5. IDEAS Center SLC VA Healthcare System, Salt Lake City, Utah, USA. 6. Division of General Internal Medicine & Public Health, Department of Medicine, Vanderbilt University School of Medicine, Nashville, Tennessee, USA. 7. Department of Biomedical Informatics, Vanderbilt University School of Medicine, Nashville, Tennessee, USA. 8. Department of Veterans Affairs Medical Center, Geriatric Research, Education and Clinical Center (GRECC), Nashville, Tennessee, USA Division of General Internal Medicine & Public Health, Department of Medicine, Vanderbilt University School of Medicine, Nashville, Tennessee, USA Department of Biostatistics, Vanderbilt University School of Medicine, Nashville, Tennessee, USA. 9. School of Biomedical Informatics, University of Texas Health Science Center, Houston, Texas, USA. 10. Department of Veterans Affairs Medical Center, Geriatric Research, Education and Clinical Center (GRECC), Nashville, Tennessee, USA Department of Biomedical Informatics, Vanderbilt University School of Medicine, Nashville, Tennessee, USA Division of General Internal Medicine & Public Health, Department of Medicine, Vanderbilt University School of Medicine, Nashville, Tennessee, USA Department of Biostatistics, Vanderbilt University School of Medicine, Nashville, Tennessee, USA.
Abstract
OBJECTIVE: To determine whether assisted annotation using interactive training can reduce the time required to annotate a clinical document corpus without introducing bias. MATERIALS AND METHODS: A tool, RapTAT, was designed to assist annotation by iteratively pre-annotating probable phrases of interest within a document, presenting the annotations to a reviewer for correction, and then using the corrected annotations for further machine learning-based training before pre-annotating subsequent documents. Annotators reviewed 404 clinical notes either manually or using RapTAT assistance for concepts related to quality of care during heart failure treatment. Notes were divided into 20 batches of 19-21 documents for iterative annotation and training. RESULTS: The number of correct RapTAT pre-annotations increased significantly and annotation time per batch decreased by ~50% over the course of annotation. Annotation rate increased from batch to batch for assisted but not manual reviewers. Pre-annotation F-measure increased from 0.5 to 0.6 to >0.80 (relative to both assisted reviewer and reference annotations) over the first three batches and more slowly thereafter. Overall inter-annotator agreement was significantly higher between RapTAT-assisted reviewers (0.89) than between manual reviewers (0.85). DISCUSSION: The tool reduced workload by decreasing the number of annotations needing to be added and helping reviewers to annotate at an increased rate. Agreement between the pre-annotations and reference standard, and agreement between the pre-annotations and assisted annotations, were similar throughout the annotation process, which suggests that pre-annotation did not introduce bias. CONCLUSIONS: Pre-annotations generated by a tool capable of interactive training can reduce the time required to create an annotated document corpus by up to 50%. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.
OBJECTIVE: To determine whether assisted annotation using interactive training can reduce the time required to annotate a clinical document corpus without introducing bias. MATERIALS AND METHODS: A tool, RapTAT, was designed to assist annotation by iteratively pre-annotating probable phrases of interest within a document, presenting the annotations to a reviewer for correction, and then using the corrected annotations for further machine learning-based training before pre-annotating subsequent documents. Annotators reviewed 404 clinical notes either manually or using RapTAT assistance for concepts related to quality of care during heart failure treatment. Notes were divided into 20 batches of 19-21 documents for iterative annotation and training. RESULTS: The number of correct RapTAT pre-annotations increased significantly and annotation time per batch decreased by ~50% over the course of annotation. Annotation rate increased from batch to batch for assisted but not manual reviewers. Pre-annotation F-measure increased from 0.5 to 0.6 to >0.80 (relative to both assisted reviewer and reference annotations) over the first three batches and more slowly thereafter. Overall inter-annotator agreement was significantly higher between RapTAT-assisted reviewers (0.89) than between manual reviewers (0.85). DISCUSSION: The tool reduced workload by decreasing the number of annotations needing to be added and helping reviewers to annotate at an increased rate. Agreement between the pre-annotations and reference standard, and agreement between the pre-annotations and assisted annotations, were similar throughout the annotation process, which suggests that pre-annotation did not introduce bias. CONCLUSIONS: Pre-annotations generated by a tool capable of interactive training can reduce the time required to create an annotated document corpus by up to 50%. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.
Entities:
Keywords:
Clinical Informatics; Guideline Adherence; Heart Failure; Medical Informatics Computing; Natural Language Processing
Authors: Michael E Matheny; Fern Fitzhenry; Theodore Speroff; Jennifer K Green; Michelle L Griffith; Eduard E Vasilevskis; Elliot M Fielstein; Peter L Elkin; Steven H Brown Journal: Int J Med Inform Date: 2012-01-12 Impact factor: 4.046
Authors: Henk Harkema; Wendy W Chapman; Melissa Saul; Evan S Dellon; Robert E Schoen; Ateev Mehrotra Journal: J Am Med Inform Assoc Date: 2011-09-21 Impact factor: 4.497
Authors: Robert O Bonow; Susan Bennett; Donald E Casey; Theodore G Ganiats; Mark A Hlatky; Marvin A Konstam; Costas T Lambrew; Sharon-Lise T Normand; Ileana L Pina; Martha J Radford; Andrew L Smith; Lynne Warner Stevenson; Gregory Burke; Kim A Eagle; Harlan M Krumholz; Jane Linderbaum; Frederick A Masoudi; James L Ritchie; John S Rumsfeld; John A Spertus Journal: Circulation Date: 2005-09-13 Impact factor: 29.690
Authors: Angus Roberts; Robert Gaizauskas; Mark Hepple; George Demetriou; Yikun Guo; Ian Roberts; Andrea Setzer Journal: J Biomed Inform Date: 2009-01-23 Impact factor: 6.317
Authors: Wendy W Chapman; Prakash M Nadkarni; Lynette Hirschman; Leonard W D'Avolio; Guergana K Savova; Ozlem Uzuner Journal: J Am Med Inform Assoc Date: 2011 Sep-Oct Impact factor: 4.497
Authors: Harvey J Murff; Fern FitzHenry; Michael E Matheny; Nancy Gentry; Kristen L Kotter; Kimberly Crimin; Robert S Dittus; Amy K Rosen; Peter L Elkin; Steven H Brown; Theodore Speroff Journal: JAMA Date: 2011-08-24 Impact factor: 56.272
Authors: John Aberdeen; Samuel Bayer; Reyyan Yeniterzi; Ben Wellner; Cheryl Clark; David Hanauer; Bradley Malin; Lynette Hirschman Journal: Int J Med Inform Date: 2010-10-14 Impact factor: 4.046
Authors: Gaurav Trivedi; Esmaeel R Dadashzadeh; Robert M Handzel; Wendy W Chapman; Shyam Visweswaran; Harry Hochheiser Journal: Appl Clin Inform Date: 2019-09-04 Impact factor: 2.342
Authors: Youngjun Kim; Jennifer H Garvin; Mary K Goldstein; Tammy S Hwang; Andrew Redd; Dan Bolton; Paul A Heidenreich; Stéphane M Meystre Journal: J Biomed Inform Date: 2017-02-02 Impact factor: 6.317
Authors: Kavishwar B Wagholikar; Christina M Fischer; Alyssa Goodson; Christopher D Herrick; Martin Rees; Eloy Toscano; Calum A MacRae; Benjamin M Scirica; Akshay S Desai; Shawn N Murphy Journal: J Med Syst Date: 2018-09-25 Impact factor: 4.460