Muhammad F Walji1, Elsbeth Kalenderian2, Mark Piotrowski3, Duong Tran3, Krishna K Kookal3, Oluwabunmi Tokede2, Joel M White4, Ram Vaderhobli4, Rachel Ramoni5, Paul C Stark6, Nicole S Kimmes7, Maxim Lagerweij8, Vimla L Patel9. 1. The University of Texas Health Science Center at Houston, United States. Electronic address: Muhammad.F.Walji@uth.tmc.edu. 2. Harvard School of Dental Medicine, United States. 3. The University of Texas Health Science Center at Houston, United States. 4. University of California, San Francisco, United States. 5. Harvard Medical School, United States. 6. Tufts University, United States. 7. Creighton University, United States. 8. Academic Centre for Dentistry at Amsterdam (ACTA), United States. 9. The New York Academy of Medicine, United States.
Abstract
OBJECTIVE: To comparatively evaluate the effectiveness of three different methods involving end-users for detecting usability problems in an EHR: user testing, semi-structured interviews and surveys. MATERIALS AND METHODS: Data were collected at two major urban dental schools from faculty, residents and dental students to assess the usability of a dental EHR for developing a treatment plan. These included user testing (N=32), semi-structured interviews (N=36), and surveys (N=35). RESULTS: The three methods together identified a total of 187 usability violations: 54% via user testing, 28% via the semi-structured interview and 18% from the survey method, with modest overlap. These usability problems were classified into 24 problem themes in 3 broad categories. User testing covered the broadest range of themes (83%), followed by the interview (63%) and survey (29%) methods. DISCUSSION: Multiple evaluation methods provide a comprehensive approach to identifying EHR usability challenges and specific problems. The three methods were found to be complementary, and thus each can provide unique insights for software enhancement. Interview and survey methods were found not to be sufficient by themselves, but when used in conjunction with the user testing method, they provided a comprehensive evaluation of the EHR. CONCLUSION: We recommend using a multi-method approach when testing the usability of health information technology because it provides a more comprehensive picture of usability challenges.
OBJECTIVE: To comparatively evaluate the effectiveness of three different methods involving end-users for detecting usability problems in an EHR: user testing, semi-structured interviews and surveys. MATERIALS AND METHODS: Data were collected at two major urban dental schools from faculty, residents and dental students to assess the usability of a dental EHR for developing a treatment plan. These included user testing (N=32), semi-structured interviews (N=36), and surveys (N=35). RESULTS: The three methods together identified a total of 187 usability violations: 54% via user testing, 28% via the semi-structured interview and 18% from the survey method, with modest overlap. These usability problems were classified into 24 problem themes in 3 broad categories. User testing covered the broadest range of themes (83%), followed by the interview (63%) and survey (29%) methods. DISCUSSION: Multiple evaluation methods provide a comprehensive approach to identifying EHR usability challenges and specific problems. The three methods were found to be complementary, and thus each can provide unique insights for software enhancement. Interview and survey methods were found not to be sufficient by themselves, but when used in conjunction with the user testing method, they provided a comprehensive evaluation of the EHR. CONCLUSION: We recommend using a multi-method approach when testing the usability of health information technology because it provides a more comprehensive picture of usability challenges.
Authors: Emily M Campbell; Dean F Sittig; Joan S Ash; Kenneth P Guappone; Richard H Dykstra Journal: J Am Med Inform Assoc Date: 2006-06-23 Impact factor: 4.497
Authors: Johanna Viitanen; Hannele Hyppönen; Tinja Lääveri; Jukka Vänskä; Jarmo Reponen; Ilkka Winblad Journal: Int J Med Inform Date: 2011-07-23 Impact factor: 4.046
Authors: Ross Koppel; Joshua P Metlay; Abigail Cohen; Brian Abaluck; A Russell Localio; Stephen E Kimmel; Brian L Strom Journal: JAMA Date: 2005-03-09 Impact factor: 56.272
Authors: Blackford Middleton; Meryl Bloomrosen; Mark A Dente; Bill Hashmat; Ross Koppel; J Marc Overhage; Thomas H Payne; S Trent Rosenbloom; Charlotte Weaver; Jiajie Zhang Journal: J Am Med Inform Assoc Date: 2013-01-25 Impact factor: 4.497
Authors: José Carlos Ferrão; Mónica Duarte Oliveira; Filipe Janela; Henrique M G Martins Journal: Appl Clin Inform Date: 2016-12-07 Impact factor: 2.342
Authors: Belén Cruz Zapata; Antonio Hernández Niñirola; Ali Idri; José Luis Fernández-Alemán; Ambrosio Toval Journal: J Med Syst Date: 2014-06-24 Impact factor: 4.460
Authors: Elizabeth Cutting; Meghan Banchero; Amber L Beitelshees; James J Cimino; Guilherme Del Fiol; Ayse P Gurses; Mark A Hoffman; Linda Jo Bone Jeng; Kensaku Kawamoto; Mark Kelemen; Harold Alan Pincus; Alan R Shuldiner; Marc S Williams; Toni I Pollin; Casey Lynnette Overby Journal: J Biomed Inform Date: 2016-07-14 Impact factor: 6.317
Authors: Aarti Bhardwaj; Rachel Ramoni; Elsbeth Kalenderian; Ana Neumann; Nutan B Hebballi; Joel M White; Lyle McClellan; Muhammad F Walji Journal: J Am Dent Assoc Date: 2015-11-06 Impact factor: 3.634