Marijke Broekhuis1, Lex van Velsen2, Hermie Hermens2. 1. Roessingh Research and Development, Roessinghsbleekweg 33b, 7522AH, Enschede, the Netherlands; Biomedical Signals and Systems, Faculty of Electrical Engineering, Mathematics and Computer Science (EEMCS), University of Twente, Enschede, the Netherlands. Electronic address: m.broekhuis@rrd.nl. 2. Roessingh Research and Development, Roessinghsbleekweg 33b, 7522AH, Enschede, the Netherlands; Biomedical Signals and Systems, Faculty of Electrical Engineering, Mathematics and Computer Science (EEMCS), University of Twente, Enschede, the Netherlands.
Abstract
BACKGROUND: It is generally assumed that usability benchmarking instruments are technology agnostic. The same methods for usability evaluations are used for digital commercial, educational, governmental and healthcare systems. However, eHealth technologies have unique characteristics. They need to support patients' health, provide treatment or monitor progress. Little research is done on the effectiveness of different benchmarks (qualitative and quantitative) within the eHealth context. OBJECTIVES: In this study, we compared three usability benchmarking instruments (logging task performance, think aloud and the SUS, the System Usability Scale) to assess which metric is most indicative of usability in an eHealth technology. Also, we analyzed how these outcome variables (task completion, system usability score, serious and critical usability issues) interacted with the acceptance factors Perceived benefits, Usefulness and Intention to use. METHODS: A usability evaluation protocol was set up that incorporated all three benchmarking methods. This protocol was deployed among 36 Dutch participants and across three different eHealth technologies: a gamified application for older adults (N = 19), an online tele-rehabilitation portal for healthcare professionals (N = 9), and a mobile health app for adolescents (N = 8). RESULTS: The main finding was that task completion, compared to the SUS, had stronger correlations with usability benchmarks. Also, serious and critical issues were stronger correlated to task metrics than the SUS. With regard to acceptance factors, there were no significant differences between the three usability benchmarking instruments. CONCLUSIONS: With this study, we took a first step in examining how to improve usability evaluations for eHealth. The results show that listing usability issues from think aloud protocols remains one of the most effective tools to explain the usability for eHealth. Using the SUS as a stand-alone usability metric for eHealth is not recommended. Preferably, the SUS should be combined with task metrics, especially task completion. We recommend to develop a usability benchmarking instrument specifically for eHealth.
BACKGROUND: It is generally assumed that usability benchmarking instruments are technology agnostic. The same methods for usability evaluations are used for digital commercial, educational, governmental and healthcare systems. However, eHealth technologies have unique characteristics. They need to support patients' health, provide treatment or monitor progress. Little research is done on the effectiveness of different benchmarks (qualitative and quantitative) within the eHealth context. OBJECTIVES: In this study, we compared three usability benchmarking instruments (logging task performance, think aloud and the SUS, the System Usability Scale) to assess which metric is most indicative of usability in an eHealth technology. Also, we analyzed how these outcome variables (task completion, system usability score, serious and critical usability issues) interacted with the acceptance factors Perceived benefits, Usefulness and Intention to use. METHODS: A usability evaluation protocol was set up that incorporated all three benchmarking methods. This protocol was deployed among 36 Dutch participants and across three different eHealth technologies: a gamified application for older adults (N = 19), an online tele-rehabilitation portal for healthcare professionals (N = 9), and a mobile health app for adolescents (N = 8). RESULTS: The main finding was that task completion, compared to the SUS, had stronger correlations with usability benchmarks. Also, serious and critical issues were stronger correlated to task metrics than the SUS. With regard to acceptance factors, there were no significant differences between the three usability benchmarking instruments. CONCLUSIONS: With this study, we took a first step in examining how to improve usability evaluations for eHealth. The results show that listing usability issues from think aloud protocols remains one of the most effective tools to explain the usability for eHealth. Using the SUS as a stand-alone usability metric for eHealth is not recommended. Preferably, the SUS should be combined with task metrics, especially task completion. We recommend to develop a usability benchmarking instrument specifically for eHealth.
Authors: Allard W Olthof; Anne L M Leusveld; Jan Cees de Groot; Petra M C Callenbach; Peter M A van Ooijen Journal: J Med Syst Date: 2020-07-28 Impact factor: 4.460
Authors: Inocencio Daniel Maramba; Ray Jones; Daniela Austin; Katie Edwards; Edward Meinert; Arunangsu Chatterjee Journal: JMIR Med Inform Date: 2022-03-29
Authors: Stefan Sigle; Pilar Barriga; Francisco Javier Correa Fernández; Christian Juhra; Steffen Härtel; Christian Fegeler Journal: JMIR Mhealth Uhealth Date: 2020-06-03 Impact factor: 4.773
Authors: Melissa T Baysari; Mai H Duong; Patrick Hooper; Michaela Stockey-Bridge; Selvana Awad; Wu Yi Zheng; Sarah N Hilmer Journal: BMC Med Inform Decis Mak Date: 2021-04-05 Impact factor: 2.796
Authors: Tim Pelle; Job van der Palen; Frank de Graaf; Frank H J van den Hoogen; Karen Bevers; Cornelia H M van den Ende Journal: BMC Health Serv Res Date: 2021-05-10 Impact factor: 2.655
Authors: Tessy Boedt; Nele Steenackers; Joke Verbeke; Astrid Vermeulen; Charlotte De Backer; Peter Yiga; Christophe Matthys Journal: Front Nutr Date: 2022-01-21