Effect of a Novel Clinical Decision Support Tool on the Efficiency and Accuracy of Treatment Recommendations for Cholesterol Management.

BACKGROUND: The 2013 American College of Cardiology / American Heart Association Guidelines for the Treatment of Blood Cholesterol emphasize treatment based on cardiovascular risk. But finding time in a primary care visit to manually calculate cardiovascular risk and prescribe treatment based on risk is challenging. We developed an informatics-based clinical decision support tool, MayoExpertAdvisor, to deliver automated cardiovascular risk scores and guideline-based treatment recommendations based on patient-specific data in the electronic heath record.
OBJECTIVE: To assess the impact of our clinical decision support tool on the efficiency and accuracy of clinician calculation of cardiovascular risk and its effect on the delivery of guideline-consistent treatment recommendations.
METHODS: Clinicians were asked to review the EHR records of selected patients. We evaluated the amount of time and the number of clicks and keystrokes needed to calculate cardiovascular risk and provide a treatment recommendation with and without our clinical decision support tool. We also compared the treatment recommendation arrived at by clinicians with and without the use of our tool to those recommended by the guidelines.
RESULTS: Clinicians saved 3 minutes and 38 seconds in completing both tasks with MayoExpertAdvisor, used 94 fewer clicks and 23 fewer key strokes, and improved accuracy from the baseline of 60.61% to 100% for both the risk score calculation and guideline-consistent treatment recommendation.
CONCLUSION: Informatics solution can greatly improve the efficiency and accuracy of individualized treatment recommendations and have the potential to increase guideline compliance.

1. Background And Significance

Guideline compliance for the prevention and treatment of atherosclerotic cardiovascular disease (ASCVD) is suboptimal, even in the highest risk patients. This was true under the Adult Treatment Panel III (ATP III) guidelines in 2001 [1] and seems to be continuing under the 2013 American College of Cardiology (ACC) and the American Heart Association (AHA) guidelines [2]. This is despite the easy availability web-based individual cardiovascular risk score calculators such as the ACC risk calculator (tools.acc.org/ASCVD-Risk-Estimator/) to aid clinicians in calculating risk. To use such a calculator a clinician must manually enter data such as age, sex, race, blood pressure, hypertension history, smoking status, and cholesterol level.

One factor that may be contributing to the under treatment of high risk patients is the time and click burden for primary care clinicians to complete this calculation and determine the appropriate medication. One can presume from other observations of EHR navigation that this may be challenging [3-4]. The literature also suggests that there is inadequate time in a typical visit to gather information necessary for such a risk calculation [5-6]. Primary care also must address preventive services and chronic disease management. The U.S. Preventive Services Task Force estimated that a clinician would need 7.4 hours each day just to comply with guidelines for preventive services [6]. Another barrier is the current state of the EHR, containing an overwhelming amount of information in the clinical and structured data elements, not necessarily optimized for clinician workflow [3-4]. Significant time is spent interacting with the EHR trying to find relevant information and complete clinical documentation [7-8]. Further, information overload and the presentation of data in the health record may increase the number of clinical errors [9-11].

In order to automate the process of calculating risk and determining appropriate treatment, and hopefully thus improve the prevention and treatment of ASCVD, we have developed a clinical decision support tool, MayoExpertAdvisor (MEA), that provides cardiovascular risk scores and guideline-based treatment recommendations based on patient-specific data in the EHR. In this study, we aim to show that our clinical decision support tool improves the efficiency and accuracy of clinician calculation of cardiovascular risk and increases the rate at which clinicians provide guideline-consistent treatment recommendations.

Although similar tools to automate the delivery of cardiovascular risk scores and treatment recommendations and have been developed elsewhere [12-13], to our knowledge ours is the first to undergo a formal evaluation to test its effect on the efficiency and accuracy of clinician treatment recommendations. Additionally, our tool is likely the first to use data derived through natural language processing (NLP) and the first to present treatment recommendations alongside supporting data and links to educational resources and decision aids.

2. Objectives

The objective of this study is to assess the impact of MEA on the efficiency and accuracy of clinician calculation of cardiovascular risk as well as MEA’s effect on the delivery of guideline-consistent treatment recommendations.

3. Methods

3.1 Practice Setting

The clinicians invited to participate in this study were from Employee and Community Health and Cardiology in Mayo Clinic, Rochester, Minnesota. Thirty-three clinicians of the eligible 262 including 13 primary care internal medicine physicians, 4 family medicine physicians, 10 internal medicine residents, and 6 cardiology nurse practitioners participated in the study.

3.2 Patient Selection

Real patient data was used in this study to simulate MEA use in clinical practice. The patients in the study were selected from the Employee and Community Health population which consists of patients receiving primary care at Mayo Clinic, Rochester, MN.

We randomly selected patients who were not on the recommended dose of statin, determined by manual review. We looked at two groups of patients. The first group included 409 patients between the ages of 40 and 75 with an LDL greater than 190mg/dL and no history of ASCVD. The second group included 3880 diabetic patients between the ages of 40 and 75 with an LDL over 70 mg/dL or Non-HDL over 100 and no history of ASCVD. We selected four patients randomly from each group, and these 8 patients were used for the test scenarios in the study.

3.3 MEA System Description

MEA provides care recommendations that are established by care process models (CPMs) in AskMayoExpert, an internal resource designed for point of care knowledge delivery. The knowledge of the CPMs is based on the most recent guidelines and input from subject matter experts. The CPMs are presented as flow-charts so that clinicians may proceed through each decision point until they arrive at the appropriate care recommendation. Embedded within the CPMs are risk score calculators and links to relevant patient education materials. Our CPM for the management of cholesterol differs only minimally from the recommendations of the 2013 ACC/AHA guidelines [2].

The clinical data needed to automate the care processes come from diverse sources including multiple EHR and internal specialty specific applications (▶ Figure 1). Both structured data and unstructured data concepts are extracted from the clinical notes via natural language processing (NLP). In order to collect and deliver the data in one service layer, a unified data platform (UDP) has been developed at Mayo Clinic. The UDP serves as a composite data source that is needed to execute the rules for the MEA initiative.

Fig. 1

Architecture Diagram of MEA System

The identification of data needed at each node in the CPMs and the clinical logic portrayed therein was translated into programmable logic utilizing a collaborative effort between clinicians, informaticians, and technologists. Using C#.NET, this logic was encoded into a rules engine.

A web service for the ACC/AHA ASCVD risk calculator is prefilled with data from the EHR to determine the risk for a given patient. The user interface provides a risk score when MEA is accessed (▶ Figure 2). The user interface also allows users to modify the data parameters in the calculator and see the effect on patient risk for ASCVD without affecting data in the EHR (▶ Figure 3).

Fig. 2

MayoExpertAdvisor User Interface: Interface includes care recommendation, risk score, and relevant patient data.

Fig. 3

Risk Calculator Interface: Risk calculator pulls data from EHR and allows the user to do “what if” scenarios without affecting the data in the EHR.

Finally the system delivers individualized care recommendations based on patient-specific data from the EHR and the knowledge from the CPMs (▶ Figure 2). In addition to the recommended clinical action, the user interface also delivers automated risk score calculations and access to relevant patient education materials and shared decision-making tools. MEA is integrated into the locally produced EHR in order to limit the number of applications that clinicians need to open during patient encounters. Mayo Clinic‘s EHR is in a general electric (GE) centricity environment. To improve the user interface Mayo Clinic has developed a viewer called Synthesis that used GE web services to retrieve data and present it to clinicians in a more intuitive format. MEA has been interfaced with the Synthesis thus enabling easy navigation for the clinicians to the risk score and any recommendations regarding cholesterol management.

3.4 Study design and data collection

Each clinician was given a set of instructions, which asked them to look at two randomly selected patient records, one patient record from each group previously described. The clinicians were asked to calculate the ASCVD risk score and recommend treatment for each patient based on EHR review with or without input from MEA. The patient group for which MEA input was given was switched every other clinician so that time differences could not be attributed to different patient characteristics.

Each clinician performed the assigned tasks on a computer with Morae® software (v 3.3.3, Tech-Smith Corporation, Okemos, MI). Using Morae®, the following efficiency metrics were collected: time, number of clicks, and number of key strokes. In addition, we compared the clinicians’ ASCVD risk score calculations to the pre-calculated scores given by MEA and noted whether the care recommendation matched the suggested care on the hyperlipidemia CPM.

3.5 Survey

A survey regarding the ASCVD risk score calculator and satisfaction withthe MEA tool were administered following the completion of the study through Morae®. The survey questions are in ▶ Table 4.

Table 4

Survey of providers perception on ASCVD risk score utility

Survey of providers perception on ASCVD risk score utility and MEA prototype	Always	Most of the Time	Sometimes	Rarely	Never
1. How often would you calculate the ASCVD risk score for a given patient if it were not pre-calculated for you?	15.1%	36.4%	36.4%	12.1%	0.0%
2. Would a high 30 year risk score (>30%) influence your likelihood of initiating statin therapy if a patient‘s 10 year risk score is low? Note that the 30 year risk score applies to patients ages 20–59 years.	0.0%	27.3%	55.6%	18.2%	0.0%
3. Do you use the ASCVD risk score calculator to encourage patients to quit smoking?	15.1%	30.3%	27.3%	18.2%	9.1%
4. Do you use the ASCVD risk score calculator to encourage patients to lower blood pressure and cholesterol through exercise?	18.7%	28.1%	31.2%	15.6%	6.2%
5. I am able to trust the pre-calculated risk scores.	71.9%	25.0%	3.1%	0.0%	0.0%
6. I am able to trust the care recommendation.	59.4%	25.0%	12.5%	0.0%	3.1%
7. The care recommendations displayed on the Individualized Knowledge Page (IKP/MEA) are easy to understand.	75.8%	18.2%	6.1%	0.0%	0.0%
8. The layout of the information is logically organized for providing patient care.	69.7%	24.2%	6.1%	0.0%	0.0%

3.6 Statistics

The patient populations and data characterisics were compared using t-tests and a two-sided test with a significance < 0.05 on JMP® (v 10.0.0, SAS Instittue Inc, Cary, NC).

A t-test was used to compare the amount of time with and without assistance from MEA. ANOVA was used to compare the amount of time it took between clinician type. These tests were performed on SAS 9.3 (SAS Institute, Cary NC).

4. Results

The characteristics of the patient data used in this study is shown in ▶ Table 1. Because some patients were used more frequently in the study than others due to patient data changing during the time of the study, we confirmed that there was no selection bias by analyzing the patient data characteristic and finding no statistical differences between populations.

Table 1

Characteristics of patient data used with and without MEA

Patient Characteristic	No MEA	MEA	p value
Mean Number of medications (SD)	7.7 (4.9)	8.2 (5.4)	0.61
Document Count	170.3 (95.5)	172.4 (91.6)	0.93
Appointment Count	40.1 (22.2)	40.3 (22.0)	0.98
Hospital Days	0.1 (0.3)	0.1 (0.33)	0.71
Height Count	4.8 (2.0)	4.8 (2.0)	0.9
Weight Count	8.3 (5.1)	8.4 (5.0)	0.98
Pulse Count	10.8 (7.7)	10.7 (7.6)	0.98
BP Count	11.5 (8.7)	11.5 (8.5)	0.97
Lab Count	112.6 (89.6)	110.1 (88.0)	0.93

4.1 Time spent making calculation and recommendation

Without any assistance from MEA, the clinicians spent an average of 4 minutes and 21 seconds to calculate the ASCVD score and a total of 5 minutes and 8 seconds to additionally determine the care they would recommend for the patient. Qualitatively, some clinicians took a lot of time scrolling through previous notes to find parameters such as blood pressure rather than looking in parts of the EHR with discrete data, such as looking in the vitals section. With MEA, the clinicians spent 39 seconds to calculate the ASCVD score and a total of 1 minute and 31 seconds to calculate the ASCVD score and determine a recommendation for patient care. The clinicians saved 3 minutes and 42 seconds in calculating the ASCVD score and a total of 3 minutes and 38 seconds in determining the recommendation. The time savings were statistically significant.

A break out summary comparing the time required to complete the tasks with and without the use of MEA is found in ▶ Table 2. We analyzed statistical significance across clinician types. We found that when clinicians did not use MEA there was statistical significance (p<0.05) in the total amount of time for calculation and recommendation between nurse practitioners and physician assistants and all other clinicians types. There was also a statistically significant difference between the residents and the internal medicine physicians. However, usage of MEA resulted in no statistical difference across clinician types.

Table 2

Amount of Time to Complete Tasks

Clinician Type	Task	No MEA (in sec)			With MEA (in sec)			|Δ Time|Mean (STD)	Pr > |t|
		Mean (STD)	Min	Max	Mean (STD)	Min	Max
All Clinicians(N=33)	ASCVD Calculation	261(75)	144	465	37(19)	16	80	222(74)	<0.0001
	Total Time for Recommendation	308(101)	155	611	91(33)	49	173	218(98)	<0.0001
CNP/PA(N=6)	ASCVD Calculation	342(93)	193	465	31(8)	19	43	310(92)	0.0004
	Total Time for Recommendation	437(129)	250	611	93(17)	77	126	343(115)	0.0008
Resident(N=10)	ASCVD Calculation	197(33)	144	252	29(7)	20	47	168(35)	<0.0001
	Total Time for Recommendation	230(53)	155	317	77(29)	37	105	153(71)	<0.0001
Staff-Family Medicine(N=4)	ASCVD Calculation	286(66)	196	348	38(13)	24	46	248(66)	0.0049
	Total Time for Recommendation	314(68)	235	391	97(38)	60	137	216(32)	0.0009
Staff-Internal Medicine(N=13)	ASCVD Calculation	266(49)	182	336	50(24)	16	88	216(50)	<0.0001
	Total Time for Recommendation	307(59)	219	420	98(40)	49	173	210(67)	<0.0001

4.3 Efficiency of clicks and key strokes making calculation and recommendation

Without any assistance from MEA, clinicians have to go through the EHR clicking on multiple tabs in order to find relevant data. The quantiative nature of this time and click burden and the significantly improved the efficiency of this process is shown in ▶ Table 3.

Table 3

Efficiency Metrics

Clinician Type	Task	Mean Number (Standard Deviation)
		No MEA	With MEA	|Δ|	Pr > |t|
Clicks (N=24)	ASCVD Calculation	88(67)	1.2 (1.0)	87(67)	<0.0001
	Total for Recommendation	101(85)	6.6 (4.1)	94(84)	<0.0001
Key strokes (N=24)	ASCVD Calculation	25(36)	1.5 (4.2)	24(37)	0.0046
	Total for Recommendation	27(38)	5.2 (4.8)	22(40)	0.0132

4.4 Accuracy of calculation and recommendation

The clinicians were found to have a 60.61% accuracy of ASCVD risk score calculation and 60.61% accuracy in selecting the guideline recommended statin treatment without MEA. The clinicians had a 100% accuracy of ASCVD risk score calculation and 100% accuracy in selecting the guideline recommended statin treatment with the use of MEA. The gold-standard for ASCVD risk score and guideline-recommended treatment was the MEA output which was verified by the study team to ensure accuracy. There was no statistical difference in accuracy by type of clinician.

The most frequent errors that clinicians made in calculating the ASCVD risk score were not using the most recent blood pressure (41.7%) and incorrectly determining whether the patient was being treated for hypertension (33.3%). Clinicians also inputted the wrong gender, age, and smoking status.

4.5 Survey Results

Of the clinicians surveyed, 51.5% indicated they calculate ASCVD risk most of the time or always, and 27.3% indicated that a high 30 year risk score may affect initiating treatment. Fewer than half of the clinicians used the risk calculator to encourage patients to quit smoking (45.4%) and lower blood pressure or cholesterol (46.8%). The questions and results regarding clinician perception of ASCVD risk score utility and general satisfaction with MEA are given in ▶ Table 4.

5. Discussion

In this study we observed not only significant savings in time and efficiency, but also marked improvement in the overall accuracy for a clinician to calculate an ASCVD risk score and determine appropriate cholesterol treatment using MEA. Clinicians saved 3 minutes and 38 seconds in completing both tasks with MEA, were more efficient with 94 fewer clicks and 23 fewer key strokes with MEA, and improved accuracy from the baseline of 60.61% for both the risk score calculation and treatment recommendation. On average, there are 62 patients not on optimal cholesterol treatment seen in the Mayo Clinic primary care practice each day. If time savings is estimated only based on patients who need treatment, MEA would save the practice 3 hours and 45 minutes each day – approximately one half day of clinician time.

Providing care to patients in a timely and accurate manner is challenging when there is an overload of information [10]. We observed the types and frequency of errors that can occur with manual entry into a calculator, i.e. almost 40% of calculations were erroneous due to input of wrong information. For example, clinicians sometimes selected the wrong blood pressure reading from the EHR, or missed that a patient was on anti-hypertensive treatment, and thus chose the wrong field in the ACC risk score. With this knowledge delivery solution, data can be extracted automatically from the EHR and can populate the ACC risk score calculator to help clinicians deliver individualized treatment recommendations for patients with elevated cardiovascular risk. Only 15% of the clinicians in this study indicated that they always will calculate a patient’s ASCVD risk score and this suggests that technology can have an integral role in ensuring consistent and accurate delivery of individualized recommendations.

One of the challenges in delivering electronic knowledge recommendations from clinical guidelines has been ensuring it fits the clinician’s workflow [14]. Even though we have not yet evaluated MEA in clinical practice, the brief survey at the end of our encounter suggested a positive perception of the user interface and design (▶ Table 4). Further analysis of clincian satisfaction will be needed when MEA is implemented in practice.

An important limitation of this study is that this was conducted in a simulated clinical environment. Clinicians searched the EHR for evidence of smoking, race, etc. in calculating the ASCVD score, but in a true patient encounter, the clinician would have some of this information upon history taking and clinicians typically know patients’ medical histories reducing the amount of information that needs to be looked up. However, a clinical environment is also prone to more distractions and interruptions. Another limitation is small sample size. However, the objectively large time saving and statistical significance of these findings suggest broad applicability. Further MEA is currently only integrated useI n Mayo’s internally produced EHR viewer Synthesis. Moreover, there are plans to integrate MEA in Cerner for other Mayo Clinic sites, and to EPIC as Mayo transitions to this EHR.

6. Conclusion

Our study demonstrates opportunity for electronic knowledge delivery solutions to improve efficiency by decreasing the amount of time and the number of clicks for information retrieval required for individualized risk calculation and to increase the accuracy in the delivery of guideline-consistent treatment recommendations for reduction of cardiovascular risk by primary care clinicians.

Multiple Choice Questions

Clinical Decision Support Systems:

Can improve accuracy of calculation of individual risk scores

Can improve efficiency of clinicians by automating the task of manually entering data into scoring tools

Support clinician decision making, but allow clinicians to use their own judgments

All of the above

American College of Cardiology/American Heart Association atherosclerotic cardiovascular disease risk calculator includes which of the following:

Presence of diabetes mellitus

Systolic blood pressure

Smoking status

All of the above