Danny T Y Wu1,2, Scott Vennemeyer1, Kelly Brown3, Jason Revalee4, Paul Murdock1, Sarah Salomone1, Ashton France3, Katherine Clarke-Myers3, Samuel P Hanke2,3. 1. Department of Biomedical Informatics, University of Cincinnati, Cincinnati, Ohio, United States. 2. Department of Pediatrics, University of Cincinnati, Cincinnati, Ohio, United States. 3. Heart Institute, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, United States. 4. DAAP School of Design, University of Cincinnati, Cincinnati, Ohio, United States.
Abstract
BACKGROUND: Interactive data visualization and dashboards can be an effective way to explore meaningful patterns in large clinical data sets and to inform quality improvement initiatives. However, these interactive dashboards may have usability issues that undermine their effectiveness. These usability issues can be attributed to mismatched mental models between the designers and the users. Unfortunately, very few evaluation studies in visual analytics have specifically examined such mismatches between these two groups. OBJECTIVES: We aimed to evaluate the usability of an interactive surgical dashboard and to seek opportunities for improvement. We also aimed to provide empirical evidence to demonstrate the mismatched mental models between the designers and the users of the dashboard. METHODS: An interactive dashboard was developed in a large congenital heart center. This dashboard provides real-time, interactive access to clinical outcomes data for the surgical program. A mixed-method, two-phase study was conducted to collect user feedback. A group of designers (N = 3) and a purposeful sample of users (N = 12) were recruited. The qualitative data were analyzed thematically. The dashboards were compared using the System Usability Scale (SUS) and qualitative data. RESULTS: The participating users gave an average SUS score of 82.9 on the new dashboard and 63.5 on the existing dashboard (p = 0.006). The participants achieved high task accuracy when using the new dashboard. The qualitative analysis revealed three opportunities for improvement. The data analysis and triangulation provided empirical evidence to the mismatched mental models. CONCLUSION: We conducted a mixed-method usability study on an interactive surgical dashboard and identified areas of improvements. Our study design can be an effective and efficient way to evaluate visual analytics systems in health care. We encourage researchers and practitioners to conduct user-centered evaluation and implement education plans to mitigate potential usability challenges and increase user satisfaction and adoption. Georg Thieme Verlag KG Stuttgart · New York.
BACKGROUND: Interactive data visualization and dashboards can be an effective way to explore meaningful patterns in large clinical data sets and to inform quality improvement initiatives. However, these interactive dashboards may have usability issues that undermine their effectiveness. These usability issues can be attributed to mismatched mental models between the designers and the users. Unfortunately, very few evaluation studies in visual analytics have specifically examined such mismatches between these two groups. OBJECTIVES: We aimed to evaluate the usability of an interactive surgical dashboard and to seek opportunities for improvement. We also aimed to provide empirical evidence to demonstrate the mismatched mental models between the designers and the users of the dashboard. METHODS: An interactive dashboard was developed in a large congenital heart center. This dashboard provides real-time, interactive access to clinical outcomes data for the surgical program. A mixed-method, two-phase study was conducted to collect user feedback. A group of designers (N = 3) and a purposeful sample of users (N = 12) were recruited. The qualitative data were analyzed thematically. The dashboards were compared using the System Usability Scale (SUS) and qualitative data. RESULTS: The participating users gave an average SUS score of 82.9 on the new dashboard and 63.5 on the existing dashboard (p = 0.006). The participants achieved high task accuracy when using the new dashboard. The qualitative analysis revealed three opportunities for improvement. The data analysis and triangulation provided empirical evidence to the mismatched mental models. CONCLUSION: We conducted a mixed-method usability study on an interactive surgical dashboard and identified areas of improvements. Our study design can be an effective and efficient way to evaluate visual analytics systems in health care. We encourage researchers and practitioners to conduct user-centered evaluation and implement education plans to mitigate potential usability challenges and increase user satisfaction and adoption. Georg Thieme Verlag KG Stuttgart · New York.
Authors: Catherine Plaisant; Johnny Wu; A Zach Hettinger; Seth Powsner; Ben Shneiderman Journal: J Am Med Inform Assoc Date: 2015-02-08 Impact factor: 4.497
Authors: Danny T Y Wu; Annie T Chen; John D Manning; Gal Levy-Fix; Uba Backonja; David Borland; Jesus J Caban; Dawn W Dowding; Harry Hochheiser; Vadim Kagan; Swaminathan Kandaswamy; Manish Kumar; Alexis Nunez; Eric Pan; David Gotz Journal: J Am Med Inform Assoc Date: 2019-04-01 Impact factor: 4.497
Authors: Allan F Simpao; Luis M Ahumada; Beatriz Larru Martinez; Ana M Cardenas; Talene A Metjian; Kaede V Sullivan; Jorge A Gálvez; Bimal R Desai; Mohamed A Rehman; Jeffrey S Gerber Journal: Appl Clin Inform Date: 2018-01-17 Impact factor: 2.342
Authors: Robert P Pierce; Bernie R Eskridge; LeAnn Rehard; Brandi Ross; Margaret A Day; Jeffery L Belden Journal: Appl Clin Inform Date: 2020-09-09 Impact factor: 2.342
Authors: Jung Yin Tsang; Niels Peek; Iain Buchan; Sabine N van der Veer; Benjamin Brown Journal: J Am Med Inform Assoc Date: 2022-05-11 Impact factor: 7.942