Usability of a Digital Registry to Promote Secondary Prevention for Peripheral Artery Disease Patients.

OBJECTIVE: To evaluate usability of a quality improvement tool that promotes guideline-based care for patients with peripheral arterial disease (PAD). PATIENTS AND METHODS: The study was conducted from July 19, 2018, to August 21, 2019. We compared the usability of a PAD cohort knowledge solution (CKS) with standard management supported by an electronic health record (EHR). Two scenarios were developed for usability evaluation; the first for the PAD-CKS while the second evaluated standard EHR workflow. Providers were asked to provide opinions about the PAD-CKS tool and to generate a System Usability Scale (SUS) score. Metrics analyzed included time required, number of mouse clicks, and number of keystrokes.
RESULTS: Usability evaluations were completed by 11 providers. SUS for the PAD-CKS was excellent at 89.6. Time required to complete 21 tasks in the CKS was 4 minutes compared with 12 minutes for standard EHR workflow (median, P = .002). Completion of CKS tasks required 34 clicks compared with 148 clicks for the EHR (median, P = .002). Keystrokes for CKS task completion was 8 compared with 72 for EHR (median, P = .004). Providers indicated that overall they found the tool easy to use and the PAD mortality risk score useful.
CONCLUSIONS: Usability evaluation of the PAD-CKS tool demonstrated time savings, a high SUS score, and a reduction of mouse clicks and keystrokes for task completion compared to standard workflow using the EHR. Provider feedback regarding the strengths and weaknesses also created opportunities for iterative improvement of the PAD-CKS tool.

A growing focus on secondary prevention for peripheral artery disease (PAD) patients has emerged after the recognition of increasing prevalence and suboptimal management.1, 2, 3, 4, 5 Guideline recommended secondary prevention strategies include treatment with antiplatelet agents, statins, angiotensin-converting enzyme inhibitors (ACEIs) or angiotensin II receptor blockers (ARBs), and smoking abstention. However, many studies have shown a gap between guideline-recommended therapies and practice implementation, as most patients with PAD receive suboptimal care.,,7, 8, 9, 10 These findings suggest need for innovative solutions to assist providers in the achievement of improved care for patients with PAD.

Recently digital disease registries have been used to identify patients with a specific condition to support research and quality-improvement activities.11, 12, 13 In a collaborative effort of informaticians, clinicians, and information technologists, our institution has developed a web-based application termed cohort knowledge solution (CKS) that identifies PAD patients and summarizes clinically relevant information in a digital PAD-CKS registry. This tool aids in identifying patients with PAD by searching clinical narratives from electronic health records (EHRs) by natural language processing (NLP).,

This study aimed to evaluate the usability of a PAD-CKS tool developed to evaluate and promote adherence to guideline-based care including treatment with antiplatelet agents, statins, ACEIs or ARBs, and smoking abstention.

Methods

Development of the CKS Tool

Patients with PAD were identified by NLP, used to extract PAD status from the text of narrative clinical notes in real time using a big data-empowered clinical NLP infrastructure. The PAD-NLP algorithm is a rule-based algorithm that uses text processing, patient classification, and assertion status., It has been previously shown that NLP has greater accuracy than billing code algorithms for identification of PAD from narrative clinical notes.,

The PAD-CKS (Figure 1) displays (1) demographic information (age, sex, race, ethnicity and geocode), (2) mortality risk score estimated by automated calculator, (3) comorbidities used for calculation of risk score, (4) status of PAD guideline-recommended strategies (treatment with antiplatelet agents, statins, ACEIs or ARBs, and smoking abstention), (5) selected laboratory test results (total cholesterol, low-density lipoprotein and high-density lipoprotein cholesterol), and (6) blood pressure values (systolic and diastolic). The algorithm used to design the mortality risk calculator was derived from a community-based cohort of PAD patients from Olmsted County, Minnesota, using a Cox model for 5-year all-cause mortality. Data elements such as demographic characteristics, status of guideline recommended strategies, laboratory test results, comorbidities (using billing codes), and blood pressure values included in the PAD-CKS tool were extracted from EHR structured data supported by a unified data platform (UDP) developed at our institution. The UDP provides technical infrastructure required for consolidation, enrichment, and access of data for the PAD-CKS tool. The study was approved by Mayo Clinic Institutional Review Board and was conducted from July 19, 2018, to August 21, 2019.

Figure 1

User interface of the peripheral artery disease cohort knowledge solution tool. CAD = coronary artery disease; EGFR = estimated glomerular filtration rate; N = No; prior limb rev = prior limb revascularization procedure; Y = yes.

Development, Implementation, and Analysis of Usability

Two separate scenarios were developed to evaluate usability of the PAD-CKS tool and the EHR system; tasks for these scenarios were similar (Table 1). Tasks were created with 5 goals for assessment: (A) find patients with PAD, (B) find the status of guideline-recommended therapy, (C) find selected risk factors for mortality, (D) find relevant laboratory test results, and (E) estimate patient mortality rate (Table 1).

Table 1

Tasks Included in the Usability Test Scenarios for the EHR and CKS interfacesa

Group and Task	Description
Goal A	Find patients with PAD
Task 1	Use the electronic health record, EPIC, or the CKS tool to find the patient information.
	Review pertinent narrative clinical notes in EPIC/information in the CKS to answer the question: Has PAD been diagnosed in this patient?
Goal B	Find status of guideline-recommended therapy for PAD.
Task 2	Use the EHR/CKS to identify whether the patient is currently taking a statin drug.
Task 3	Use the EHR/CKS to identify whether the patient is currently a smoker or a nonsmoker.
Task 4	Use the EHR/CKS to identify whether the patient currently taking an antiplatelet agent.
Task 5	Use the EHR/CKS to identify whether the patient currently taking an ACEI or an ARB.
Goal C	Find selected risk factors for mortality for PAD.
Task 6	Use the EHR/CKS to identify whether CAD has been diagnosed.
Task 7	Use the EHR/CKS to identify whether dementia has been diagnosed.
Task 8	Use the EHR/CKS to identify whether type 1 or type 2 diabetes has been diagnosed.
Task 9	Use the EHR/CKS to identify whether hypertension has been diagnosed.
Task 10	Use the EHR/CKS to identify whether the patient has had a prior limb revascularization.
Task 11	Use the EHR/CKS to identify whether renal disease has been diagnosed.
Goal D	Find relevant laboratory test results.
Task 12	Use the EHR/CKS to identify the most recent creatinine level.
Task 13	Use the EHR/CKS to identify the most recent e-GFR.
Task 14	Use the EHR/CKS to identify the most recent hemoglobin A1C level.
Task 15	Use the EHR/CKS to identify most recent total cholesterol levels.
Task 16	Use the EHR/CKS to identify most recent LDL levels.
Task 17	Use the EHR/CKS to identify most recent AST levels.
Task 18	Use the EHR/CKS to identify most recent ALT levels.
Task 19	Use the EHR/CKS to identify most recent fasting blood glucose.
Task 20	Use the EHR/CKS to identify most recent triglyceride levels.
Goal E	Estimate patient risk for mortality
Task 21	Use the patient information provided in the EHR/CKS to rate the patient mortality risk within the next 5 years as high, medium, or low.

ACEI = angiotensin-converting enzyme inhibitor; ALT = alanine aminotransferase; ARB = angiotensin-receptor blocker; AST = aspartate aminotransferase; CAD = coronary artery disease; CKS = cohort knowledge solution; e-GFR = estimated glomerular filtration rate; EHR = electronic health record; LDL = low-density lipoprotein; PAD = peripheral artery disease.

Morae usability software was used to run usability tests and to analyze collected data. Each scenario was followed by user rating of ease of completion of tasks on a scale from 1 to 5, where 1 was very difficult to complete and 5 was very easy to complete. Ease of completion ratings were averaged to enable comparison of the CKS and standard EHR workflow. Providers also completed the System Usability Scale (SUS)20, 21, 22 for the CKS. SUS is a 10-question questionnaire with 5 response options for measuring usability. The 5 response options range from “strongly agree” to “strongly disagree.”20, 21, 22 Participants were asked to score the following 10 questions of the SUS questionnaire: (1) I think that I would like to use this system frequently; (2) I found the system unnecessarily complex; (3) I thought the system was easy to use; (4) I think that I would need the support of a technical person to be able to use this system; (5) I found the various functions in this system were well integrated; (6) I thought there was too much inconsistency in this system; (7) I would imagine that most people would learn to use this system very quickly; (8) I found the system very cumbersome to use; (9) I felt very confident using the system; and (10) I needed to learn a lot of things before I could get going with this system. SUS scores could range from 0 to 100. Scores greater than 70 are considered acceptable or good, whereas scores of 85 or greater indicate high level of usability or excellent score., Scores of 50 or less indicate poor or unacceptable usability., Providers were asked to provide opinions whether positive, neutral, or negative about the PAD-CKS tool. Verbal feedback was recorded by Morae software and transcribed for qualitative analysis.

An initial test run was conducted using 2 providers familiar with the PAD-CKS tool to evaluate functionalities relevant to the study questions and excluded from final analysis. Eleven providers were selected from the Division of Primary Care of the Department of Internal Medicine, or the Department of Cardiovascular Medicine including 6 licensed physicians, 2 nurse practitioners, 1 registered nurse, and 2 pharmacists to participate in 1-hour test sessions. Ten providers completed both the usability tests in the EHR and CKS. One provider completed only the CKS.

Data collected from the usability test included CKS SUS scores,, verbal feedback, ease of scenario completion, time required to complete each task, number of clicks for each task, and number of keystrokes for each task. Median values for number of clicks, time required, and number of keystrokes were used to summarize findings owing to skewed distribution of data. Means were calculated for ease of completion rating and SUS score, as data distributions of these variables had normal distribution. A nonparametric signed-rank test was used to evaluate P values for each category.

Results

The mean SUS score of the PAD-CKS tool was 89.6, consistent with excellent system usability. Qualitative feedback by providers with suggestions for improvements are summarized in Table 2. Major comments included requests for (1) display of origin of data elements used in the CKS, (2) more information about risk score calculation, and (3) display of all information to avoid need for scrolling. Favorable qualitative feedback about the tool is summarized in Table 3. Providers indicated that overall they found the tool easy to use and the mortality risk score useful.

Table 2

Transcription From Verbal Feedback: Potential Modifications for PAD-CKS Toola

Summary	Provider comments
Display origin of data elements in the PAD-CKS	“It would be nice to have a column specifically indicating PAD.”
	“Where is it capturing information from?”
	“Is the patient information displayed up to date? “
More information about risk mortality score calculation	“It is not clear to me if it is a mortality risk score by years or within one year.”
	“I am unsure of the criteria of the mortality risk score.”
Display all information to avoid need for scrolling	“I think it would be much better if you don’t have to scroll down as much.”
	“It would be better if you could open the patient and have all the patient information displayed at once instead of scrolling to see the information.”
	“I would use it frequently if I didn’t have to scroll as much.”

CKS = cohort knowledge solution; PAD = peripheral artery disease.

Table 3

Transcription From Verbal Feedback: Favorable Feedback for PAD-CKS Toola

Summary	Provider comments
Overall structure of the tool	“I like how the labs are situated together.”
	“I like how it has the medicines together and the disease states together. I think that’s helpful because you’re looking at it kind of all chunked chronologically.”
	“I like that the date the patient is put on a statin is also included.”
Risk mortality score	“The risk mortality score is really nice to have in here.”
Overall ease of use	“This is super user friendly and quick, I can just look at all of it by going down the row.”

CKS = cohort knowledge solution; PAD = peripheral artery disease.

For the CKS, each of the 21 tasks had an average ease of completion rating greater than 4, where 1 was very difficult to complete and 5 was very easy to complete. Eight of 21 tasks had an average rating of 5 for the CKS usability tests. The task with the lowest average ease of completion for the CKS was Task #2 (use EHR/CKS to identify whether the patient is currently taking a statin drug). By comparison, within the EHR none of 21 tasks had an average rating of 5 and only 6 had an average rating of 4 or greater. The task with the lowest average ease of completion in the EHR was Task #7 (use EHR/CKS to identify whether patient has been diagnosed with dementia). The greatest difference for the average ease of completion rating between the CKS and the EHR was for Task #7.

Total time required to complete 21 tasks in the CKS was 4 minutes compared with 12 minutes for the EHR (median, P = .002; Table 4). Completion of CKS tasks required 34 clicks compared with 148 clicks for the EHR (median, P = .002; Table 4). For the CKS, 8 keystrokes were required to complete task scenarios, whereas the EHR required 72 keystrokes (median, P = .004; Table 4).

Table 4

Time, Number of Clicks, and Number of Key-strokes for Tasks Within EHR and CKSa,b

	Time required, minutes	Number of clicks	Number of keystrokes
EHR	12.0 (8.4, 30.5)	148 (67, 3503)	72 (6, 163)
CKS	4.0 (2.3, 6.7)	34 (12, 149)	8 (1, 13)
EHR-CKS differencec	7.9 (5.2, 27.8)d	128 (36, 3354)e	66 (–7, 154)f

CKS = cohort knowledge solution; EHR = electronic health record.

Data summarized as median (minimun, maximum) across the 21 tasks based on 11 providers who used CKS and 10 who used EHR.

Difference based on data from 10 providers who used both systems.

P = .002.

P = .002.

P = .004.

Table 5 shows results for each task, and Table 6 displays accumulated results for each goal. In the CKS, Task 21 had the greatest median duration (29.6 seconds), and Task 1 had the greatest number of key strokes (n = 8) and number of mouse clicks (n = 4; Table 5). In the EHR, Task 6 had the greatest duration (72.2 seconds), Task 1 had the highest number of key strokes (n = 7), and Tasks 2 and 6 had the highest number of mouse clicks at (n = 10.5; Table 5). Goal B had the greatest median duration in CKS (81.1 seconds), Goal A had the greatest number of key strokes (n = 8), and Goal D had the greatest number of mouse clicks (n = 9) (Table 6). In the EHR, Goal C had the greatest median duration (n = 267.4 seconds). Goal D had the greatest number of key strokes (n = 21) and number of mouse clicks (n = 48.5; Table 6).

Table 5

Results for each taska

Task	CKS			EHR
	Median duration (seconds)	Median number of key strokes	Median number of mouse clicks	Median duration (seconds)	Median number of key strokes	Median number of mouse clicks
1	23.5	8.0	4.0	31.8	7.0	3.5
2	19.6	0.0	1.0	40.3	1.0	10.5
3	13.0	0.0	1.0	25.8	0.0	7.5
4	7.4	0.0	0.0	56.5	0.0	7.5
5	11.7	0.0	1.0	17.8	0.0	2.0
6	7.4	0.0	1.0	72.2	0.5	10.5
7	4.7	0.0	0.0	53.5	2.5	6.5
8	5.3	0.0	1.0	18.1	0.0	6.0
9	6.7	0.0	0.0	9.7	0.0	1.0
10	4.0	0.0	0.0	26.5	0.5	5.0
11	8.3	0.0	1.0	27.7	0.0	3.0
12	12.0	0.0	1.0	6.5	0.0	1.0
13	4.4	0.0	0.0	5.6	0.0	0.0
14	7.6	0.0	1.0	60.7	4.0	5.0
15	8.8	0.0	1.0	20.1	3.0	6.0
16	6.7	0.0	1.0	4.8	0.0	1.0
17	6.9	0.0	1.0	35.4	0.0	5.0
18	5.9	0.0	1.0	11.3	0.0	1.0
19	4.9	0.0	0.0	27.4	0.0	3.5
20	6.3	0.0	1.0	9.8	0.0	2.0
21	29.6	0.0	3.0	45.0	2.0	3.0

CKS = cohort knowledge system; EHR = electronic health record.

Table 6

Accumulated Results for Each Goala

Goal	CKS			EHR
	Median duration (seconds)	Median number of key strokes	Median number of mouse clicks	Median duration (seconds)	Median number of key strokes	Median number of mouse clicks
A	23.5	8.0	4.0	31.8	7.0	3.5
B	81.1	0.0	6.0	173.7	2.0	42.5
C	35.9	0.0	4.0	267.4	7.5	41.0
D	68.0	0.0	9.0	244.2	21.0	48.5
E	29.6	0.0	3.0	45.0	2.0	3.0

CKS = cohort knowledge system; EHR = electronic health record.

Discussion

The major findings of this study were the high acceptance rate of the CKS tool by providers and the decreased time for providers to complete tasks relating to patient care compared with standard workflow using the EHR. These observations were demonstrated by decreased number of mouse clicks and keystrokes, greater ease in completion of tasks, a lower completion time, and high SUS scores for the CKS tool. This study also demonstrated that the CKS tool saved time for each patient encounter. Information obtained from transcribed verbal feedback from users also provides valuable information regarding which functionalities of the tool can be improved.

To our knowledge, this is the first study to evaluate usability of a digital disease registry for a research and quality improvement project. Prior studies have evaluated the usability of other health information technologies including EHRs,, secure messaging for clinical document sharing, and mobile health. Although other studies have described methodologies to assemble digital disease registries from EHR data for patients with psoriasis, trophoblastic disease, and cancer who underwent radiation therapy, the usability of these registries was not evaluated.

In the present study, an area for improvement was the request to inform providers how PAD patients were identified. Considering this feedback, we have added a “Sentence with Evidence” column to the tool displaying concepts identified by NLP as well as sentences from clinical narratives where these concepts were processed. Example of concepts and sentences extracted by NLP included: (1) lower extremity, peripheral arterial disease (“gentleman with medical comorbidities significant for diabetes mellitus type 2, atherosclerotic left lower extremity peripheral arterial disease, tobacco use, and stage 3 chronic kidney disease”) and (2) leg, bypass (“male with atrial fibrillation on Coumadin, peripheral artery disease status post fem pop bypass, presented with mechanical fall”). This functionality is unique to this system because it extracts information automatically from clinical narratives by NLP. Another major comment reported by the providers was dissatisfaction with the scrolling mechanism of the tool. To address this concern, the user interface will be modified to display relevant information at the single patient view replacing the current multi-patient view.

Verbal feedback from providers also suggested that information about how the tool works should be provided before engagement with the CKS. To address this concern, a simple user instruction guide for the PAD-CKS tool will be created. This guide will address the questions “Where is it capturing information from?” and “Is the patient information displayed up to date?” The instructions will state that the CKS tool extracts up-to-date patient information from EHR data daily enabled by the UDP using the big data and real-time NLP infrastructure of the EHR. We will also add instructions regarding the risk estimator addressing the concerns “It is not clear to me if it is a mortality risk score by years or within one year” and “I am unsure of the criteria of the mortality risk score.”

The CKS tool can transform clinical practice by bringing the right information to the right provider efficiently enabling practice standardization. The CKS tool is also generalizable to other diseases. However, disease-specific algorithms to populate the tool automatically should be developed and evaluated before implementation and dissemination.

Conclusion

Usability evaluation of the PAD-CKS tool demonstrated time-savings, a high SUS score, and a reduction of mouse clicks and keystrokes for task completion compared to standard workflow using the EHR. Provider feedback regarding the strengths and weaknesses also created opportunities for iterative improvement of the PAD-CKS tool.

We will use these research observations to develop a more user-friendly PAD-CKS tool. The PAD-CKS will expedite the translation of guideline-recommended strategies into real-world practice settings by displaying individualized information summarizing use of guideline recommended strategies by each patient which will support point-of-care discussion to promote adherence to these strategies.