Safiya Richardson1, Rebecca Mishuris2, Alexander O'Connell3, David Feldstein4, Rachel Hess5, Paul Smith6, Lauren McCullagh7, Thomas McGinn8, Devin Mann9. 1. Hofstra Northwell School of Medicine, United States. Electronic address: srichard12@northwell.edu. 2. Boston University, United States. Electronic address: rebecca.mishuris@bmc.org. 3. Hofstra Northwell School of Medicine, United States. Electronic address: aoconnel@northwell.edu. 4. University of Wisconsin School of Medicine and Public Health, United States. Electronic address: df2@medicine.wisc.edu. 5. University of Utah School of Medicine, United States. Electronic address: rachel.hess@hsc.utah.edu. 6. University of Wisconsin School of Medicine and Public Health, United States. Electronic address: paul.smith@fammed.wisc.edu. 7. Hofstra Northwell School of Medicine, United States. Electronic address: lmccullagh@northwell.edu. 8. Hofstra Northwell School of Medicine, United States. Electronic address: tmcginn@northwell.edu. 9. New York University School of Medicine, United States. Electronic address: devin.mann@nyumc.org.
Abstract
OBJECTIVES: Low provider adoption continues to be a significant barrier to realizing the potential of clinical decision support. "Think Aloud" and "Near Live" usability testing were conducted on two clinical decision support tools. Each was composed of an alert, a clinical prediction rule which estimated risk of either group A Streptococcus pharyngitis or pneumonia and an automatic order set based on risk. The objective of this study was to further understanding of the facilitators of usability and to evaluate the types of additional information gained from proceeding to "Near Live" testing after completing "Think Aloud". METHODS: This was a qualitative observational study conducted at a large academic health care system with 12 primary care providers. During "Think Aloud" testing, participants were provided with written clinical scenarios and asked to verbalize their thought process while interacting with the tool. During "Near Live" testing participants interacted with a mock patient. Morae usability software was used to record full screen capture and audio during every session. Participant comments were placed into coding categories and analyzed for generalizable themes. Themes were compared across usability methods. RESULTS: "Think Aloud" and "Near Live" usability testing generated similar themes under the coding categories visibility, workflow, content, understand-ability and navigation. However, they generated significantly different themes under the coding categories usability, practical usefulness and medical usefulness. During both types of testing participants found the tool easier to use when important text was distinct in its appearance, alerts were passive and appropriately timed, content was up to date, language was clear and simple, and each component of the tool included obvious indicators of next steps. Participant comments reflected higher expectations for usability and usefulness during "Near Live" testing. For example, visit aids, such as automatically generated order sets, were felt to be less useful during "Near-Live" testing because they would not be all inclusive for the visit. CONCLUSIONS: These complementary types of usability testing generated unique and generalizable insights. Feedback during "Think Aloud" testing primarily helped to improve the tools' ease of use. The additional feedback from "Near Live" testing, which mimics a real clinical encounter, was helpful for eliciting key barriers and facilitators to provider workflow and adoption.
OBJECTIVES: Low provider adoption continues to be a significant barrier to realizing the potential of clinical decision support. "Think Aloud" and "Near Live" usability testing were conducted on two clinical decision support tools. Each was composed of an alert, a clinical prediction rule which estimated risk of either group A Streptococcus pharyngitis or pneumonia and an automatic order set based on risk. The objective of this study was to further understanding of the facilitators of usability and to evaluate the types of additional information gained from proceeding to "Near Live" testing after completing "Think Aloud". METHODS: This was a qualitative observational study conducted at a large academic health care system with 12 primary care providers. During "Think Aloud" testing, participants were provided with written clinical scenarios and asked to verbalize their thought process while interacting with the tool. During "Near Live" testing participants interacted with a mock patient. Morae usability software was used to record full screen capture and audio during every session. Participant comments were placed into coding categories and analyzed for generalizable themes. Themes were compared across usability methods. RESULTS: "Think Aloud" and "Near Live" usability testing generated similar themes under the coding categories visibility, workflow, content, understand-ability and navigation. However, they generated significantly different themes under the coding categories usability, practical usefulness and medical usefulness. During both types of testing participants found the tool easier to use when important text was distinct in its appearance, alerts were passive and appropriately timed, content was up to date, language was clear and simple, and each component of the tool included obvious indicators of next steps. Participant comments reflected higher expectations for usability and usefulness during "Near Live" testing. For example, visit aids, such as automatically generated order sets, were felt to be less useful during "Near-Live" testing because they would not be all inclusive for the visit. CONCLUSIONS: These complementary types of usability testing generated unique and generalizable insights. Feedback during "Think Aloud" testing primarily helped to improve the tools' ease of use. The additional feedback from "Near Live" testing, which mimics a real clinical encounter, was helpful for eliciting key barriers and facilitators to provider workflow and adoption.
Authors: Emily M Campbell; Dean F Sittig; Joan S Ash; Kenneth P Guappone; Richard H Dykstra Journal: J Am Med Inform Assoc Date: 2006-06-23 Impact factor: 4.497
Authors: Timothy A D Graham; Andre W Kushniruk; Michael J Bullard; Brian R Holroyd; David P Meurer; Brian H Rowe Journal: AMIA Annu Symp Proc Date: 2008-11-06
Authors: Alice C Li; Joseph L Kannry; Andre Kushniruk; Dillon Chrimes; Thomas G McGinn; Daniel Edonyabo; Devin M Mann Journal: Int J Med Inform Date: 2012-03-27 Impact factor: 4.046
Authors: Tiffani J Bright; Anthony Wong; Ravi Dhurjati; Erin Bristow; Lori Bastian; Remy R Coeytaux; Gregory Samsa; Vic Hasselblad; John W Williams; Michael D Musty; Liz Wing; Amy S Kendrick; Gillian D Sanders; David Lobach Journal: Ann Intern Med Date: 2012-07-03 Impact factor: 25.391
Authors: David A Feldstein; Rachel Hess; Thomas McGinn; Rebecca G Mishuris; Lauren McCullagh; Paul D Smith; Michael Flynn; Joseph Palmisano; Gheorghe Doros; Devin Mann Journal: Implement Sci Date: 2017-03-14 Impact factor: 7.327
Authors: Christine Sinsky; Lacey Colligan; Ling Li; Mirela Prgomet; Sam Reynolds; Lindsey Goeders; Johanna Westbrook; Michael Tutty; George Blike Journal: Ann Intern Med Date: 2016-09-06 Impact factor: 25.391
Authors: Randall W Grout; Erika R Cheng; Aaron E Carroll; Nerissa S Bauer; Stephen M Downs Journal: Int J Med Inform Date: 2018-02-02 Impact factor: 4.046
Authors: Devin Mann; Rachel Hess; Thomas McGinn; Safiya Richardson; Simon Jones; Joseph Palmisano; Sara Kuppin Chokshi; Rebecca Mishuris; Lauren McCullagh; Linda Park; Catherine Dinh-Le; Paul Smith; David Feldstein Journal: J Gen Intern Med Date: 2020-09-01 Impact factor: 5.128
Authors: Alanna Kulchak Rahm; Nephi A Walton; Lynn K Feldman; Conner Jenkins; Troy Jenkins; Thomas N Person; Joeseph Peterson; Jonathon C Reynolds; Peter N Robinson; Makenzie A Woltz; Marc S Williams; Michael M Segal Journal: BMJ Health Care Inform Date: 2021-05
Authors: Hana Bangash; Laurie Pencille; Justin H Gundelach; Ahmed Makkawy; Joseph Sutton; Lenae Makkawy; Ozan Dikilitas; Stephen Kopecky; Robert Freimuth; Pedro J Caraballo; Iftikhar J Kullo Journal: J Pers Med Date: 2020-07-23
Authors: Lisa Hightow-Weidman; Kathryn Muessig; Kelly Knudtson; Mala Srivatsa; Ellena Lawrence; Sara LeGrand; Anna Hotten; Sybil Hosek Journal: JMIR Public Health Surveill Date: 2018-04-30