Min Soon Kim1, Michael R Aro2, Kraig J Lage2, Kevin L Ingalls2, Vivek Sindhwani2, Mia K Markey3. 1. Department of Health Management and Informatics, University of Missouri School of Medicine, Columbia, Missouri; University of Missouri Informatics Institute, Columbia, Missouri; Department of Emergency Medicine, Icahn School of Medicine at Mount Sinai, New York, New York. Electronic address: kimms@health.missouri.edu. 2. Department of Radiology, University of Missouri, Columbia, Missouri. 3. Department of Biomedical Engineering, University of Texas, Austin, Texas; Department of Imaging Physics, The University of Texas MD Anderson Cancer Center, Houston, Texas.
Abstract
PURPOSE: The objective of this study was to conduct a usability evaluation of mobile apps for supporting education and training in radiologic diagnostic decision-making processes. METHODS: Of 381 mobile apps available at two major stores (Google Play and iTunes), eight iOS apps were selected for laboratory-based usability tests. Six staff radiologists completed eight app-specific task sets, using a think-aloud strategy. The triangular methods approach included quantitative performance measures, System Usability Scale (SUS), and qualitative thematic analysis using heuristic usability principles of usability issues. RESULTS: Overall, radiologists achieved higher than 70% success, with favorable SUS scores, in completing the tasks for seven of the eight apps. However, task success rate and SUS score had a weak relation (r = 0.23), indicating that the perceived usability may not reflect the holistic usability of the app. Task analysis and self-report revealed 108 usability issues, which were condensed to 55 unique issues and categorized by nine usability themes and mapped to ten usability heuristics. Nonintuitive functionality (eg, nonintuitive or misleading labels) was the most frequent theme observed, leading to inefficient navigation. These usability findings were consistent with the 13 improvements the radiologists suggested. CONCLUSIONS: This study demonstrates the feasibility of usability evaluation of radiology mobile apps and suggests potential improvements in the development of radiology mobile apps. This study also suggests that proficiency with mobile devices may not be equivalent to being an expert user, proficient in using the apps.
PURPOSE: The objective of this study was to conduct a usability evaluation of mobile apps for supporting education and training in radiologic diagnostic decision-making processes. METHODS: Of 381 mobile apps available at two major stores (Google Play and iTunes), eight iOS apps were selected for laboratory-based usability tests. Six staff radiologists completed eight app-specific task sets, using a think-aloud strategy. The triangular methods approach included quantitative performance measures, System Usability Scale (SUS), and qualitative thematic analysis using heuristic usability principles of usability issues. RESULTS: Overall, radiologists achieved higher than 70% success, with favorable SUS scores, in completing the tasks for seven of the eight apps. However, task success rate and SUS score had a weak relation (r = 0.23), indicating that the perceived usability may not reflect the holistic usability of the app. Task analysis and self-report revealed 108 usability issues, which were condensed to 55 unique issues and categorized by nine usability themes and mapped to ten usability heuristics. Nonintuitive functionality (eg, nonintuitive or misleading labels) was the most frequent theme observed, leading to inefficient navigation. These usability findings were consistent with the 13 improvements the radiologists suggested. CONCLUSIONS: This study demonstrates the feasibility of usability evaluation of radiology mobile apps and suggests potential improvements in the development of radiology mobile apps. This study also suggests that proficiency with mobile devices may not be equivalent to being an expert user, proficient in using the apps.