Xiyu Ding1, Michael Barnett2,3, Ateev Mehrotra4,5, Delphine S Tuot6,7, Danielle S Bitterman8,9, Timothy A Miller10,11. 1. Biomedical Informatics & Data Science Section, Division of General Internal Medicine, The Johns Hopkins University School of Medicine, Baltimore, Maryland, USA. 2. Department of Health Policy and Management, Harvard T.H. Chan School of Public Health, Boston, Massachusetts, USA. 3. Division of General Internal Medicine and Primary Care, Department of Medicine, Brigham and Women's Hospital, Boston, Massachusetts, USA. 4. Department of Health Care Policy, Harvard Medical School, Boston, Massachusetts, USA. 5. Division of General Medicine, Department of Medicine, Beth Israel Deaconess Medical Center, Boston, Massachusetts, USA. 6. Division of Nephrology, Department of Medicine, University of California, San Francisco, San Francisco, California, USA. 7. UCSF Center for Innovation in Access and Quality at Zuckerberg San Francisco General Hospital, San Francisco, California, USA. 8. Department of Radiation Oncology, Brigham and Women's Hospital/Dana-Farber Cancer Institute, Boston, Massachusetts, USA. 9. Artificial Intelligence in Medicine Program, Mass General Brigham, Harvard Medical School, Boston, Massachusetts, USA. 10. Computational Health Informatics Program, Boston Children's Hospital, Boston, Massachusetts, USA. 11. Department of Pediatrics, Harvard Medical School, Boston, Massachusetts, USA.
Abstract
OBJECTIVE: Electronic consultation (eConsult) content reflects important information about referring clinician needs across an organization, but is challenging to extract. The objective of this work was to develop machine learning models for classifying eConsult questions for question type and question content. Another objective of this work was to investigate the ability to solve this task with constrained expert time resources. MATERIALS AND METHODS: Our data source is the San Francisco Health Network eConsult system, with over 700 000 deidentified questions from the years 2008-2017, from gastroenterology, urology, and neurology specialties. We develop classifiers based on Bidirectional Encoder Representations from Transformers, experimenting with multitask learning to learn when information can be shared across classifiers. We produce learning curves to understand when we may be able to reduce the amount of human labeling required. RESULTS: Multitask learning shows benefits only in the neurology-urology pair where they shared substantial similarities in the distribution of question types. Continued pretraining of models in new domains is highly effective. In the neurology-urology pair, near-peak performance is achieved with only 10% of the urology training data given all of the neurology data. DISCUSSION: Sharing information across classifier types shows little benefit, whereas sharing classifier components across specialties can help if they are similar in the balance of procedural versus cognitive patient care. CONCLUSION: We can accurately classify eConsult content with enough labeled data, but only in special cases do methods for reducing labeling effort apply. Future work should explore new learning paradigms to further reduce labeling effort.
OBJECTIVE: Electronic consultation (eConsult) content reflects important information about referring clinician needs across an organization, but is challenging to extract. The objective of this work was to develop machine learning models for classifying eConsult questions for question type and question content. Another objective of this work was to investigate the ability to solve this task with constrained expert time resources. MATERIALS AND METHODS: Our data source is the San Francisco Health Network eConsult system, with over 700 000 deidentified questions from the years 2008-2017, from gastroenterology, urology, and neurology specialties. We develop classifiers based on Bidirectional Encoder Representations from Transformers, experimenting with multitask learning to learn when information can be shared across classifiers. We produce learning curves to understand when we may be able to reduce the amount of human labeling required. RESULTS: Multitask learning shows benefits only in the neurology-urology pair where they shared substantial similarities in the distribution of question types. Continued pretraining of models in new domains is highly effective. In the neurology-urology pair, near-peak performance is achieved with only 10% of the urology training data given all of the neurology data. DISCUSSION: Sharing information across classifier types shows little benefit, whereas sharing classifier components across specialties can help if they are similar in the balance of procedural versus cognitive patient care. CONCLUSION: We can accurately classify eConsult content with enough labeled data, but only in special cases do methods for reducing labeling effort apply. Future work should explore new learning paradigms to further reduce labeling effort.
Authors: Halil Kilicoglu; Asma Ben Abacha; Yassine Mrabet; Sonya E Shooshan; Laritza Rodriguez; Kate Masterton; Dina Demner-Fushman Journal: BMC Bioinformatics Date: 2018-02-06 Impact factor: 3.169