Angad Kalra1, Amit Chakraborty2, Benjamin Fine3, Joshua Reicher4. 1. Department of Computer Science, University of Toronto, Toronto, Ontario, Canada. Electronic address: angadk@cs.toronto.edu. 2. Department of Radiology, Stanford University Hospital, Palo Alto, California. 3. Physician Lead, Department of Diagnostic Imaging, Quality and Informatics and Operational Analytics Lab, Trillium Health Partners, Mississauga, Ontario, Canada; Department of Medical Imaging, University of Toronto, Toronto, Ontario, Canada. 4. Department of Radiology, Palo Alto VA Medical Center, Palo Alto, California.
Abstract
PURPOSE: The aim of this study was to enhance multispecialty CT and MRI protocol assignment quality and efficiency through development, testing, and proposed workflow design of a natural language processing (NLP)-based machine learning classifier. METHODS: NLP-based machine learning classification models were developed using order entry input data and radiologist-assigned protocols from more than 18,000 unique CT and MRI examinations obtained during routine clinical use. k-Nearest neighbor, random forest, and deep neural network classification models were evaluated at baseline and after applying class frequency and confidence thresholding techniques. To simulate performance in real-world deployment, the model was evaluated in two operating modes in combination: automation (automated assignment of the top result) and clinical decision support (CDS; top-three protocol suggestion for clinical review). Finally, model-radiologist discordance was subjectively reviewed to guide explainability and safe use. RESULTS: Baseline protocol assignment performance achieved weighted precision of 0.757 to 0.824. Simulating real-world deployment using combined thresholding techniques, the optimized deep neural network model assigned 69% of protocols in automation mode with 95% accuracy. In the remaining 31% of cases, the model achieved 92% accuracy in CDS mode. Analysis of discordance with subspecialty radiologist labels revealed both more and less appropriate model predictions. CONCLUSIONS: A multiclass NLP-based classification algorithm was designed to drive local operational improvement in CT and MR radiology protocol assignment at subspecialist quality. The results demonstrate a simulated workflow deployment enabling automated assignment of protocols in nearly 7 of 10 cases with very few errors combined with top-three CDS for remaining cases supporting a high-quality, efficient radiology workflow.
PURPOSE: The aim of this study was to enhance multispecialty CT and MRI protocol assignment quality and efficiency through development, testing, and proposed workflow design of a natural language processing (NLP)-based machine learning classifier. METHODS: NLP-based machine learning classification models were developed using order entry input data and radiologist-assigned protocols from more than 18,000 unique CT and MRI examinations obtained during routine clinical use. k-Nearest neighbor, random forest, and deep neural network classification models were evaluated at baseline and after applying class frequency and confidence thresholding techniques. To simulate performance in real-world deployment, the model was evaluated in two operating modes in combination: automation (automated assignment of the top result) and clinical decision support (CDS; top-three protocol suggestion for clinical review). Finally, model-radiologist discordance was subjectively reviewed to guide explainability and safe use. RESULTS: Baseline protocol assignment performance achieved weighted precision of 0.757 to 0.824. Simulating real-world deployment using combined thresholding techniques, the optimized deep neural network model assigned 69% of protocols in automation mode with 95% accuracy. In the remaining 31% of cases, the model achieved 92% accuracy in CDS mode. Analysis of discordance with subspecialty radiologist labels revealed both more and less appropriate model predictions. CONCLUSIONS: A multiclass NLP-based classification algorithm was designed to drive local operational improvement in CT and MR radiology protocol assignment at subspecialist quality. The results demonstrate a simulated workflow deployment enabling automated assignment of protocols in nearly 7 of 10 cases with very few errors combined with top-three CDS for remaining cases supporting a high-quality, efficient radiology workflow.
Authors: Yasasvi Tadavarthi; Valeria Makeeva; William Wagstaff; Henry Zhan; Anna Podlasek; Neil Bhatia; Marta Heilbrun; Elizabeth Krupinski; Nabile Safdar; Imon Banerjee; Judy Gichoya; Hari Trivedi Journal: Radiol Artif Intell Date: 2022-02-02
Authors: Justus Schock; Daniel Truhn; Darius Nürnberger; Stefan Conrad; Marc Sebastian Huppertz; Sebastian Keil; Christiane Kuhl; Dorit Merhof; Sven Nebelung Journal: Sci Rep Date: 2021-12-01 Impact factor: 4.379