Usability Evaluation of NLP-PIER: A Clinical Document Search Engine for Researchers.

NLP-PIER (Natural Language Processing - Patient Information Extraction for Research) is a self-service platform with a search engine for clinical researchers to perform natural language processing (NLP) queries using clinical notes. We conducted user-centered testing of NLP-PIER's usability to inform future design decisions. Quantitative and qualitative data were analyzed. Our findings will be used to improve the usability of NLP-PIER.

Introduction

NLP-PIER was created to provide an accessible solution through a search interface to clinical researchers interested in or requiring access to clinical NLP capabilities of clinical documents [1]. The system has two interfaces: a full text search interface and a custom interface for searching Unified Medical Language System (UMLS) concepts [2]. We were interested in understanding potential design opportunities and user acceptance of NLP-PIER and to more broadly understand the needs of clinical researchers when using a self-service NLP tool.

Methods

This study was conducted at University of Minnesota (UMN) as part of its broader clinical and translational science research platform. We designed standard tasks to use NLP-PIER and asked clinical researcher particpants (n=11) to complete these tasks, usability instruments (system usability scale (SUS) [3] and NASA-Task Load Index (NASA-TLX) survey [4]), a brief interview, and exit questionnaire. Time on task, task completion percentage, and survey results were assessed. Interviews were transcribed and coded for themes

Results

For the full text search interface, questionarie scores were 69.4 (19.8) (SUS) and 18.8 (5.7) (NASA TLX) and for the Concept search interface scores were 66.1 (32.4) and 21.8 (7.7). Average time on task and task completion varied widely. In interviews, all participants expressed that NLP-PIER was easy to use and would be useful in their work.

Conclusion

End user testing of NLP-PIER identified a number of usability challenges and several solutions. Our study also demonstrated that substantial variation exists between different users. Overall, our findings illustrate the importance of incorporating user testing and feedback in the design process.