Allan Fong1, Tomilayo Komolafe2, Katharine T Adams1, Arman Cohen3, Jessica L Howe1, Raj M Ratwani1,4. 1. National Center for Human Factors in Healthcare, Washington, District of Columbia, United States. 2. Virginia Polytechnic Institute and State University, Blacksburg, Virginia, United States. 3. Allen Institute for Artificial Intelligence, Seattle, Washington, United States. 4. Georgetown University Medical Center, Washington, District of Columbia, United States.
Abstract
BACKGROUND: With the pervasive use of health information technology (HIT) there has been increased concern over the usability and safety of this technology. Identifying HIT usability and safety hazards, mitigating those hazards to prevent patient harm, and using this knowledge to improve future HIT systems are critical to advancing health care. PURPOSE: The purpose of this work is to demonstrate the feasibility of a modeling approach to identify HIT usability-related patient safety events (PSEs) from the free-text of safety reports and the utility of such models for supporting patient safety analysts in their analysis of event data. METHODS: We evaluated three feature representations (bag-of-words [BOWs], topic modeling, and document embeddings) to classify HIT usability-related PSE reports using 5,911 manually annotated reports. Model results were reviewed with patient safety analysts to gather feedback on their usefulness and integration into workflow. RESULTS: The combination of term frequency-inverse document frequency BOWs and document embedding features modeled with support vector machine (SVM) with radial basis function (RBF) had the highest overall precision-recall area under the curve (AUC) and f1-score, 72 and 66%, respectively. Using only document embedding features achieved a similar precision-recall AUC and f1-score performance with the SVM RBF model, 70 and 66%, respectively. Models generally favored specificity and sensitivity over precision. Patient safety analysts found the model results to be useful and offered three suggestions on how it can be integrated into their workflow at the point of report entry, in a visual dashboard layer, and to support data retrievals. CONCLUSION: Text mining and document embeddings can support identification of HIT usability-related PSE reports. The positive feedback received on the HIT usability model shows its potential utility in real-world applications. Georg Thieme Verlag KG Stuttgart · New York.
BACKGROUND: With the pervasive use of health information technology (HIT) there has been increased concern over the usability and safety of this technology. Identifying HIT usability and safety hazards, mitigating those hazards to prevent patient harm, and using this knowledge to improve future HIT systems are critical to advancing health care. PURPOSE: The purpose of this work is to demonstrate the feasibility of a modeling approach to identify HIT usability-related patient safety events (PSEs) from the free-text of safety reports and the utility of such models for supporting patient safety analysts in their analysis of event data. METHODS: We evaluated three feature representations (bag-of-words [BOWs], topic modeling, and document embeddings) to classify HIT usability-related PSE reports using 5,911 manually annotated reports. Model results were reviewed with patient safety analysts to gather feedback on their usefulness and integration into workflow. RESULTS: The combination of term frequency-inverse document frequency BOWs and document embedding features modeled with support vector machine (SVM) with radial basis function (RBF) had the highest overall precision-recall area under the curve (AUC) and f1-score, 72 and 66%, respectively. Using only document embedding features achieved a similar precision-recall AUC and f1-score performance with the SVM RBF model, 70 and 66%, respectively. Models generally favored specificity and sensitivity over precision. Patient safety analysts found the model results to be useful and offered three suggestions on how it can be integrated into their workflow at the point of report entry, in a visual dashboard layer, and to support data retrievals. CONCLUSION: Text mining and document embeddings can support identification of HIT usability-related PSE reports. The positive feedback received on the HIT usability model shows its potential utility in real-world applications. Georg Thieme Verlag KG Stuttgart · New York.
Authors: Raj M Ratwani; Erica Savage; Amy Will; Allan Fong; Dean Karavite; Naveen Muthu; A Joy Rivera; Cori Gibson; Don Asmonga; Ben Moscovitch; Robert Grundmeier; Josh Rising Journal: Health Aff (Millwood) Date: 2018-11 Impact factor: 6.301
Authors: Allan Fong; Nicole Harriott; Donna M Walters; Hanan Foley; Richard Morrissey; Raj R Ratwani Journal: Int J Med Inform Date: 2017-05-11 Impact factor: 4.046
Authors: Mary G Amato; Alejandra Salazar; Thu-Trang T Hickman; Arbor Jl Quist; Lynn A Volk; Adam Wright; Dustin McEvoy; William L Galanter; Ross Koppel; Beverly Loudin; Jason Adelman; John D McGreevey; David H Smith; David W Bates; Gordon D Schiff Journal: J Am Med Inform Assoc Date: 2017-03-01 Impact factor: 4.497
Authors: Qian Yang; John Zimmerman; Aaron Steinfeld; Lisa Carey; James F Antaki Journal: ACM Trans Comput Hum Interact Date: 2016-05 Impact factor: 2.351
Authors: Robert P Pierce; Bernie R Eskridge; LeAnn Rehard; Brandi Ross; Margaret A Day; Jeffery L Belden Journal: Appl Clin Inform Date: 2020-09-09 Impact factor: 2.342