Errors in Electronic Health Record-Based Data Query of Statin Prescriptions in Patients With Coronary Artery Disease in a Large, Academic, Multispecialty Clinic Practice.

BACKGROUND: With the recent implementation of the Medicare Quality Payment Program, providers face increasing accountability for delivering high-quality care. Such pay-for-performance programs aim to leverage systematic data captured by electronic health record (EHR) systems to measure performance; however, the fidelity of EHR query for assessing performance has not been validated compared with manual chart review. We sought to determine whether our institution's methodology of EHR query could accurately identify cases in which providers failed to prescribe statins for eligible patients with coronary artery disease. METHODS AND
RESULTS: A total of 9459 patients with coronary artery disease were seen at least twice at the Emory Clinic between July 2014 and June 2015, of whom 1338 (14.1%, 95% confidence interval 13.5-14.9%) had no statin prescription or exemption per EHR query. A total of 120 patient cases were randomly selected and reviewed by 2 physicians for further adjudication. Of the 120 cases initially classified as statin prescription failures, only 21 (17.5%; 95% confidence interval, 11.7-25.3%) represented true failure following physician review.
CONCLUSIONS: Sole reliance on EHR data query to measure quality metrics may lead to significant errors in assessing provider performance. Institutions should be cognizant of these potential sources of error, provide support to medical providers, and form collaborative data management teams to promote and improve meaningful use of EHRs. We propose actionable steps to improve the accuracy of EHR data query that require hypothesis testing and prospective validation in future studies.

Clinical Perspective

What Is New?

Programs such as the Medicare Quality Payment Program aim to leverage electronic health records (EHRs) in holding healthcare providers accountable for delivering high‐quality care; however, the fidelity of querying EHR data to accurately measure provider performance is not well established.

Query of our EHR system identified 1338 of 9459 (14.1%) outpatients with coronary artery disease in a 1‐year period who had no detectable statin prescription or exemption (ie, were deemed a metric “failure”), yet physician adjudication of 120 randomly sampled performance “failure” cases revealed that 82.5% of patient cases were misclassified as a metric failure.

What Are the Clinical Implications?

Automated query of our EHR system was prone to error when attempting to identify eligible patients with coronary artery disease for whom appropriate statin therapy was not prescribed.

Review of patient cases that were erroneously classified as performance failures demonstrated multifactorial root causes for misclassification, each requiring different strategies to improve the fidelity of automated quality performance measurements.

This initiative highlights the possible risks of sole reliance on automated EHR data query in measuring performance without appropriate safeguards.

Healthcare providers have become increasingly reliant on electronic health record (EHR) systems.1, 2 At the same time, programs such as Meaningful Use propose to leverage EHR systems to document care processes, evaluate individual and practice performance, and ultimately improve patient outcomes.3, 4, 5, 6 Indeed, querying population‐level EHR data has the potential to be a highly efficient way for practices and insurers to gauge performance and adherence to performance measures. Nevertheless, recent studies suggest that EHR data query may have limited fidelity compared with manual chart review.7, 8 For this reason, it is critical that providers, healthcare institutions, and policy makers understand the limitations and common errors that result from trying to measure quality performance through EHR data query.9 This is especially important in light of ongoing programs such as Medicare's Physician Quality Reporting System (PQRS)10 and the recently implemented Medicare Quality Payment Program (a result of the Medicare Access and CHIP Reauthorization Act, or MACRA, legislation),11, 12 which ultimately aim to leverage EHR systems to link providers’ adherence to performance measures to their reimbursement.13, 14, 15, 16

Performance measures are particularly well defined in the context of atherosclerotic coronary artery disease (CAD). Statin medications specifically have been identified as a high‐value therapy for patients with atherosclerotic CAD17, 18, 19 and thus have received a class I recommendation in clinical practice guidelines.20 Such strong evidence for treatment can serve as an objective metric by which providers are held accountable for quality performance.21, 22 However, despite increasing pressure to adhere to quality guidelines,11, 12, 13, 14, 15 contemporary registries have illustrated suboptimal rates of appropriate statin therapy for eligible patients.23, 24, 25, 26, 27 Although this gap in care is multifactorial, a study published >10 years ago found that errors in EHR data query undermined the accurate identification patients with clinical CAD who were on optimal medical therapy.28 Now, in the Meaningful Use era, and after a decade of continued widespread use of EHR systems, we sought to compare the accuracy of EHR data query with manual chart review in determining the percentage of eligible patients with clinical CAD who were not prescribed statin therapy.

Methods

The scope of this initiative was reviewed with Emory's institutional review board before publication. Our institutional review board determined that this initiative did not require the board's oversight because the study did not constitute human subjects research but instead qualified as a quality improvement initiative. The data, analytic methods, and study materials will not be made available to other researchers for the purposes of reproducing the results or replicating the procedure because it would require access to our institution's EHRs.

Study Population

This initiative was conducted at the Emory Clinic, an academic clinical practice that is the largest multispecialty practice in the state of Georgia. The Emory Clinic employs >2000 physician and nonphysician providers (including nurse practitioners and physician assistants) and accommodates between 2500 000 and 3000 000 patient encounters annually at >30 clinical locations throughout Georgia. This initiative was undertaken as a quality improvement project aimed at improving the rate of provider prescriptions of statins for eligible patients with CAD at our institution. A waiver from our institutional review board was obtained.

This initiative included patients aged ≥18 years with ≥2 outpatient encounters between July 1, 2014 to June 30, 2015 at primary and subspecialty clinics, with a diagnosis of CAD as identified by International Classification of Diseases, Ninth Revision (ICD‐9), Current Procedural Terminology (CPT), and Systematized Nomenclature of Medicine (SNOMED) codes (Figure S1). Outpatients could be seen by either a physician or a nonphysician provider (nurse practitioner or physician assistant) from one of the following clinic sections: general internal medicine, gerontology, endocrinology, nephrology, obstetrics/gynecology, pulmonology, gastroenterology, rheumatology, infectious disease, family medicine, cardiology, and cardiac outreach. We counted patient encounters from these clinical sections because Medicare's PQRS program includes these same sections for its CAD metrics for large clinic practices (as part of the Group Practice Reporting Option [GPRO]).29

Study Procedures

We queried the Emory Clinic's clinical data warehouse (CDW), which is a repository of data from the Emory Clinic's EHR system (Cerner). The CDW extracts information nightly from Cerner Millennium. This information includes demographics, claims with diagnosis codes, orders, nursing documentation, clinical notes, procedures, imaging results, laboratory values, medication lists, allergies, visit summaries, and ancillary results, among other clinical subject areas. The data are integrated with billing and administrative data from a variety of other sources. Administrative data were used to derive a diagnosis of CAD from ICD‐9, CPT, and SNOMED billing codes as well as baseline characteristics of the patient population. Clinical data derived from the CDW were used to identify patients who were prescribed a statin or who had a documented statin allergy, intolerance, or refusal. Minimal data cleansing is performed, beyond integration with the administrative data. The expectation is that the CDW exactly reflects what can be viewed directly in Cerner with the exception that any data documented only within the free‐text portion of the provider notes were not readily extracted. The CDW resides on a standard Oracle database in a custom‐designed dimensional model. Routine access is through our business intelligence platform MicroStrategy, which was used to pull the information for this initiative.

Query of our EHR data identified 9459 unique patients representing a total of 29 713 outpatient encounters seen for a diagnosis of CAD and no statin exemption during the selected period. Consistent with Medicare's GPRO guidelines, patients had to be seen at least twice during a 12‐month period to be included in the performance analysis. Diagnosis codes (ICD‐9) and procedure codes (CPT) were used to identify baseline characteristics of the population (Figure S2). Statin prescription was measured by querying the EHR active medication list, which identifies any active orders for medications, along with any documented home or self‐reported medications. Emory mandates that all providers use its electronic prescribing system. Providers are encouraged to perform medicine reconciliation at each visit so that any medications prescribed outside of the Emory Clinic can be documented for the purposes of data capture. Query of EHR data was further used to identify patients with a statin exemption if they had a statin allergy, intolerance, or medication refusal listed in the allergy section or problem list. EHR data query identified 1338 (14.1%) patients with CAD and no documentation of statin therapy or exemption.30 These query parameters were established in accordance with Medicare's 2015 GPRO guidelines, which calculates performance using the following equation (equation 1):29

Of the 1338 cases classified by EHR data query as statin metric “failures,” 120 unique patients were identified by stratified random sampling for manual physician adjudication. Forty charts were randomly selected from each of the 3 clinical sections with the highest numbers of patient encounters for CAD: general internal medicine, cardiology, and cardiac outreach clinic. We also verified that the 120 cases all represented unique patients so as to avoid double‐counting individual patients. The number of sampled charts (n=120) was determined based on what was deemed feasible within a reasonable time frame for the purposes of our quality improvement project.

Each of the 120 patient charts was reviewed independently by 2 physicians. The reviewers assessed each patient's problem list, medications, and allergies. Reviewers also assessed provider notes, procedural documentation, diagnostic results, and outpatient visit summaries for the 3 most recent visits during the reporting period. The primary objective of chart review was to determine whether providers had failed to prescribe statin therapy in eligible patients with CAD, in accordance with the 2013 American College of Cardiology/American Heart Association (ACC/AHA) Task Force Guidelines for the Treatment of Blood Cholesterol to Reduce Atherosclerotic Cardiovascular Risk in Adults.20 Specifically, we utilized the 2013 ACC/AHA cholesterol guideline definition of coronary atherosclerotic cardiovascular disease (ASCVD), which includes history of acute myocardial infarction, stable or unstable angina, or coronary revascularization presumed to be of atherosclerotic origin to determine whether each patient met criteria for a class I recommendation of statin use.20 During chart review, each of the following questions was asked sequentially to determine whether the provider had failed to prescribe statin therapy for eligible patients with CAD, according to the 2013 ACC/AHA cholesterol guidelines (Figure) 20: (1) Did the patient meet criteria for “clinical” CAD and, therefore, have a class I indication for statin prescription? (2) Was the patient prescribed a statin according to the EHR medication list? (3) Was an exemption from statin therapy documented in the EHR allergies section or problem list (which represent discrete data elements in our CDW)? (4) Was an exemption from statin therapy documented in the provider's free‐text note during clinical encounters and thus not be captured during EHR data query? (5) Did the patient have a comorbidity such as end‐stage renal disease on dialysis or heart failure with New York Heart Association (NYHA) class II–IV symptoms for which the 2013 ACC/AHA cholesterol guidelines declined recommendations for or against statin use in patients with coronary ASCVD?20 If these criteria were not satisfied (Figure), reviewers adjudicated the case as a true failure of providers to prescribe statin therapy (ie, a true gap in clinical care). If the 2 initial reviewers disagreed, they discussed the cases to achieve consensus, and if the disagreement persisted, a third physician reviewer provided a tie‐break.

Figure 1

Algorithm for determining whether EHR data query accurately detected statin prescription failures in patients with CAD. ACC indicates American College of Cardiology; AHA, American Heart Association; ASCVD, atherosclerotic cardiovascular disease; CAD, coronary artery disease; EHR, electronic health record; ESRD, end‐stage renal disease; HF, heart failure; NYHA, New York Heart Association.

We also performed a sensitivity analysis to determine whether inclusion of older adults affected the rate of EHR‐based classification of provider performance. We excluded patients aged >75 years, for whom the recommendation for high‐intensity statins for coronary ASCVD is reduced from a class I to a class IIa strength of recommendation, depending on provider concerns about adverse effects, patient preference, polypharmacy, and drug–drug interaction.20, 31, 32

Statistical Analysis

Confidence intervals (CIs) at 95% were determined as a proportion without correction for continuity. An unweighted Cohen's κ coefficient was determined by classifying each of the 2 reviewers’ adjudications of the selected cases into a true versus erroneous designation of quality performance failure and categorizing the disagreement in a 2×2 matrix.33

Results

Patient Characteristics

Query of our EHR data identified a total of 9459 unique patients seen across 29 713 outpatient encounters for CAD without documented statin exemption between July 1, 2014, and July 30, 2015. Baseline characteristics of this patient population are summarized (Table 1, Figure S2).

Table 1

Baseline Characteristics of the Patients With Presumed CAD Based on EHR Query Seen at the Emory Clinic July 1, 2014 to June 30, 2015

Characteristic	Patients in All Clinical Sections of the Emory Clinic (9459)	Patients Within the 3 Clinical Subsections of Cardiology, Cardiac Outreach, and GIM (8938)	Patients Randomly Selected for Review (120)
Age, y, mean±SD	74±11.6	73.6±11.5	74.4 ±12.4
Female, n (%)	3472 (36.7)	3294 (36.9)	49 (40.8)
Race, n (%)
White	6107 (64.7)	5818 (65.1)	85 (70.8)
Black	2808 (29.7)	2624 (29.4)	28 (23.3)
Asian	166 (1.8)	152 (1.7)	0 (0)
Hispanic or Latino	118 (1.3)	106 (1.2)	2 (1.6)
Other	239 (2.5)	216 (2.4)	0 (0)
Race data missing	21 (0.2)	11 (0.9)	1 (0.8)
Prior history of MI, n (%)	586 (6.2)	555 (6.2)	4 (3.3)
Prior history of PCI, n (%)	2584 (27.4)	2499 (28.0)	19 (15.8)
Prior history of CABG, n (%)	1243 (13.2)	1168 (13.1)	7 (5.8)
Diabetes mellitus, n (%)	3113 (33.0)	2864 (32.0)	33 (27.5)
Hypertension, n (%)	7775 (82.4)	7372 (82.5)	95 (79.2)
Heart failure, n (%)	2041 (21.6)	1949 (21.8)	31 (25.83)

Baseline characteristics were identified by ICD‐9 and CPT codes (see Figure S2 for full list of specific ICD‐9 and CPT codes used). CABG indicates coronary artery bypass grafting; CAD, coronary artery disease; CPT, Current Procedural Terminology; EHR, electronic health record; GIM, general internal medicine; ICD‐9, International Classification of Diseases, Ninth Revision; MI, myocardial infarction; PCI, percutaneous coronary intervention.

Performance According to EHR Data Query

Between July 1, 2014, and June 30, 2015, EHR data query identified that 8121 of 9459 patients (85.9%; 95% CI, 85.1–86.5%) with CAD had a documented statin prescription or exemption. The statin prescription rates among the 3 specific clinical sections with the highest numbers of outpatient encounters for CAD (and for which manual chart review was performed) were as follows: cardiac outreach, 2621 of 3051 (85.9%; 95% CI, 84.6–87.1%); general cardiology, 3999 of 4562 (87.7%; 95% CI, 86.7–88.6%); and general internal medicine, 1090 of 1325 (82.3%; 95% CI, 80.1–84.2%). Baseline characteristics of the patient population from these 3 sections are shown in Table 1.

Performance According to Manual Chart Review by Physicians

EHR data query classified 1338 of 9459 patients (14.1%; 95% CI, 13.5–14.9%) with CAD as statin failures (ie, no statin prescribed and no documented statin exemption). From these 1338 “failure” cases, 120 unique patients (including 40 from each of the 3 sections of cardiac outreach, general cardiology, and general internal medicine) were adjudicated by 2 physicians (baseline characteristics shown in Table 1). Of these 120 patients, only 21 (17.5%; 95% CI, 11.7–25.3%) represented a true failure of the provider to prescribe statin therapy for eligible patients with CAD (or a true gap in clinical care). The remaining 99 patients (82.5%; 95% CI, 74.7–88.3%) were misclassified as cases in which the provider failed to meet the statin performance metric in eligible patients with CAD (Table 2). Fifty‐five (45.8%; 95% CI, 37.2–54.7%) of these patients did not meet criteria for coronary ASCVD; of those, 18 (15.0%; 95% CI, 9.7–22.5%) underwent left heart catheterization and had no evidence of obstructive disease, 14 (11.7%; 95% CI, 7.1–18.6%) were inappropriately diagnosed with clinical CAD solely based on a high coronary artery calcium score, 12 (10.0%; 95% CI, 5.8–16.7%) did not have any diagnostic code consistent with coronary ASCVD on problem list, and the remaining 11 (9.2%; 95% CI, 5.2–15.7%) had CAD on the problem list but either were clinically asymptomatic or underwent negative cardiac stress testing. The remaining 44 patients (36.7%; 95% CI, 28.6–45.6%) did meet criteria for coronary ASCVD but still did not have a true gap in clinical care (Table 2): For 19 patients (15.8%; 95% CI, 10.4–23.4%), a statin exemption was documented within the free‐text clinic visit note but not in a discrete EHR field; for 9 patients (7.5%; 95% CI, 4.0–13.6%), EHR data query did not detect a statin exemption listed in a discrete EHR field (ie, allergies tab or problem list); for 12 patients (10.0%; 95% CI, 5.8–16.7%), EHR data query did not detect an active statin prescription on the medication list; and 4 patients (3.3%; 95% CI, 1.3–8.3%) had a diagnosis of heart failure with NYHA class II–IV symptoms.

Table 2

Results of Physician Adjudication of 120 Cases Sampled From Patients With Presumed CAD Without a Statin Prescription or Documented Exemption

No Clinical ASCVD of Coronary Origina (n=55, 45.8%)	EHR Data Query Did Not Detect Patient on Statin (n=12, 10.0%)	EHR Data Query Did Not Detect Statin Exemption (n=9, 7.5%)	Exemption to Statin Listed in Free‐Text Note Only (n=19, 15.8%)	Comorbidity Downgrades Strength of Evidence for Statin (n=4, 3.3%)	True Failure to Prescribe Statin (n=21, 17.5%)
CAD on problem list, but patient had coronary angiogram with no evidence of obstructive CAD (n=18, 15.0%)	Simvastatin (n=4, 3.3%)	Allergy listed to “statins” (n=7, 5.8%)	Patient intolerant to statins NOS (n=7, 5.8%)	Heart failure with NYHA class II–IV symptoms (n=4, 3.3%)	…
CAD on problem list, but patient had only CT scan showing coronary artery calcium (n=14, 11.7%)	Pravastatin (n=3, 2.5%)	Allergy listed to “simvastatin” (n=2, 1.7%)	Patient with liver disease or elevated LFTs (n=7, 5.8%)	…	…
CAD or coronary ASCVD equivalent code not on problem list (n=12, 10.0%)	Atorvastatin (n=3, 2.5%)	…	Patient refused statin (n=3, 2.5%)	…	…
CAD on problem list, but patient had noninvasive stress testing, ruling out ischemic etiology of chest pain (n=7, 5.0%)	Rosuvastatin (n=2, 1.7%)	…	Statin not prescribed because of medication interaction or side effect (n=2, 1.7%)	…	…
CAD on EHR problem list, but patient did not have clinical symptoms and no other diagnostic testing was performed (n=4, 3.3%)	…	…	…	…	…

ACC indicates American College of Cardiology; AHA, American Heart Association; ASCVD, atherosclerotic cardiovascular disease; CAD, coronary artery disease; CT, computed tomography; EHR, electronic health record; LFTs, liver function tests; NOS, not otherwise specified; NYHA, New York Heart Association.

The 2013 ACC/AHA cholesterol guidelines define clinical ASCVD of coronary origin as a history of acute myocardial infarction, stable or unstable angina, or coronary revascularization presumed to be of atherosclerotic origin.

Cohen's κ coefficient was calculated to determine agreement between independent reviewers during manual chart review. The value of κ was 0.863 (SE: 0.059; 95% CI, 0.747–0.980), which is considered very good agreement. For cases in which there was initial disagreement between the 2 reviewers, the 2 reviewers discussed these cases and achieved consensus in all cases.

Sensitivity Analysis of Statin Use in CAD

A sensitivity analysis was performed to exclude patients aged >75 years, for whom use of high‐intensity statins for CAD is less well established.20 Of the 120 total cases reviewed, 69 patients (57.5%) were aged ≤75 years. Twelve of these 69 patients (17.4%; 95% CI, 10.2–30.0%) met criteria for coronary ASCVD but lacked documentation for statin therapy or acceptable exemption and thus were adjudicated as having a true gap in clinical care. Fifty‐seven patients (82.6%; 95% CI, 72.0–89.8%) were misclassified by EHR data query as statin performance failures; of those, 38 (55.1%; 95% CI, 43.4–66.2%) did not meet criteria for coronary ASCVD (Table 3), whereas 19 (27.5%; 95% CI, 18.4–39.1%) had documented clinical CAD. Of those 19 with documented clinical CAD, 9 (13.0%; 95% CI, 7.0–23.0%) had a statin exemption written in the provider's free‐text clinical note, 3 (4.4%; 95% CI, 1.5–12.0%) had a documented statin exemption listed in the allergy section or problem list that was missed by EHR data query, 5 (7.3%; 95% CI, 3.1–15.9%) were on a statin according to the medication list, and 2 (2.9%; 95% CI, 0.8–10.0%) had heart failure with NYHA class II–IV symptoms (Table 3).

Table 3

Results of Physician Adjudication of 69 Patients From Sensitivity Analysis of Patients Aged ≤75 Years, Sampled From Patients With Presumed CAD Without a Statin Prescription or Documented Exemption

No Clinical ASCVD of Coronary Origina (n=38, 55.1%)	EHR Data Query Did Not Detect Patient on Statin (n=5, 7.2%)	EHR Data Query Did Not Detect Statin Exemption (n=3, 4.4%)	Exemption to Statin Listed in Free‐Text Note Only (n=9, 13.0%)	Comorbidity Downgrades Strength of Evidence for Statin (n=2, 2.9%)	True Failure to Prescribe Statin (n=12, 17.4%)
CAD on problem list, but patient had coronary angiogram with no evidence of obstructive CAD (n=10, 14.5%)	Simvastatin (n=2, 2.9%)	Allergy listed to “statins” (n=3, 4.4%)	Patient intolerant to statins NOS (n=3, 4.4%)	Heart failure with NYHA class II–IV symptoms (n=2, 2.9%)	…
CAD on problem list, but patient had only CT scan showing coronary artery calcium (n=10, 14.5%)	Pravastatin (n=1, 1.4%)	…	Patient with liver disease or elevated LFTs (n=4, 5.8%)	…	…
CAD or coronary ASCVD equivalent code not on problem list (n=9, 13.0%)	Atorvastatin (n=1, 1.4%)	…	Patient refused statin (n=1, 1.4%)	…	…
CAD on problem list, but patient had noninvasive stress testing, ruling out ischemic etiology of chest pain (n=7, 10.1%)	Rosuvastatin (n=1, 1.4%)	…	Statin not prescribed because of medication interaction or side effect (n=1, 1.4%)	…	…
CAD on EHR problem list, but patient lacked clinical symptoms and no other diagnostic testing was performed (n=2, 2.9%)	…	…	…	…	…

ACC indicates American College of Cardiology; AHA, American Heart Association; ASCVD, atherosclerotic cardiovascular disease; CAD, coronary artery disease; CT, computed tomography; EHR, electronic health record; LFTs, liver function tests; NOS, not otherwise specified; NYHA, New York Heart Association.

The 2013 ACC/AHA cholesterol guidelines define clinical ASCVD of coronary origin as a history of acute myocardial infarction, stable or unstable angina, or coronary revascularization presumed to be of atherosclerotic origin.

Discussion

The aim of our initiative was to validate whether EHR data query at our institution had appropriately classified cases in which eligible patients with CAD had been prescribed a statin by their medical provider. Query of our institution's CDW identified a 14.1% rate of failure to prescribe statins for eligible patients with CAD; however, only 17.5% of these cases were correctly classified as performance failures compared with manual chart adjudication, suggesting that our institution's EHR data query had falsely identified a substantial number of patients failing the statin metric. The proportion of patients correctly classified as performance failures was similar (17.4%) when we excluded older patients who did not satisfy criteria for a class I recommendation for statin therapy. Our findings are consistent with a prior study published >10 years ago28 that reported EHR data query was highly prone to errors in identifying cases in which providers failed to prescribe lipid‐lowering therapy in patients with CAD. Over the past decade, there has been increasing reliance on the use of EHRs and systematic data query; however, despite this fact, the medical field and EHR vendors have failed to validate or even significantly improve EHR‐based data capture as a reliable way to measure performance.

In our initiative, there were 4 major reasons for errors of classification by automated EHR‐based query. The most common cause of classification error was ambiguity between the umbrella term CAD versus the most recent guideline definition of coronary ASCVD. Although most clinicians agree that <50% lesions on left heart catheterization or coronary artery calcifications represent an early spectrum of coronary ASCVD requiring further risk stratification and modification, these patients do not necessarily satisfy the criteria for a class 1 recommendation for statin therapy on the basis of clinically manifest coronary ASCVD alone. The next most common reason for erroneous classification was due to the clinical provider not appropriately revising the EHR face sheet to reflect clinical data. This includes cases in which the provider listed a statin exemption in the free‐text portion of the note only, which prevented detection by automated data query. It also includes cases in which providers did not revise or remove ASCVD‐equivalent diagnoses from the problem list if the patient was either asymptomatic or underwent negative stress testing. The third most common reason was related to the EHR detection algorithm not incorporating appropriate inclusion or exclusion criteria. This includes patients for whom the provider listed a statin exemption or prescription in a discrete EHR data field. The authors determined that the detection algorithm was overly strict in these cases because if a patient had been discontinued from a statin at any point during the review period, regardless of whether he or she had been on a statin for the majority of the year or even restarted on a statin during the review period, the patient would have been classified as not being on a statin for the year. As for statin exemptions, the Medicare GPRO specifications dictated that it was not merely sufficient to list “statins” on the allergy list. If the “reaction” to statins in the allergy list was specified as “side effect” rather than “allergy” or “intolerance,” the CDW detection algorithm did not classify this as a proper statin exemption. In addition, the EHR data query algorithm included patients with NYHA class II–IV heart failure who would be exempt from statin therapy, as per AHA/ACC guidelines. Finally, the fourth most common reason for erroneous classification was integration of administrative billing data (Table 4).

Table 4

Classification of Errors and Proposed Actionable Steps Identified During Physician‐Adjudicated Review of Automated Data Query in Determining Quality Performance for Statin Prescriptions in Eligible Patients With CAD

Type of Error	Disparate Definitions of CAD Between Clinical Consensus Guidelines and Standardized Coding Terminology	Clinical Provider Did Not Appropriately Revise EHR Face Sheet Given Clinical Data	EHR Detection Algorithm Did Not Incorporate Appropriate Inclusion/Exclusion Criteria	Integration of Administrative Billing Data
Example of error	CAD on problem list, but patient had coronary angiogram with no evidence of obstructive CAD; CAD on problem list, but patient had only CT scan showing coronary artery calcium	CAD on problem list, but patient had noninvasive stress testing, ruling out ischemic etiology of chest pain; CAD on EHR problem list, but patient lacked clinical symptoms and no other diagnostic testing was performed; exemption to statin listed in free‐text note only	EHR data query did not detect patient on statin; EHR data query did not detect statin exemption; comorbidity downgrades strength of evidence for statin	CAD or coronary ASCVDa equivalent code not on problem list
Actionable step to reduce error	Revise diagnostic coding schemes to match clinical guideline diagnoses that call for specific guideline‐recommended treatments	Engineer alerts into the EHR notifying providers that the patient has a diagnostic code eligible for assessment of quality performance—and prompt a response	Providers should work with data analytic teams to revise detection algorithms to ensure proper inclusion/exclusion criteria adjusted through performance year	Discontinue integration of administrative billing data

ACC indicates American College of Cardiology; AHA, American Heart Association; ASCVD, atherosclerotic cardiovascular disease; CAD, coronary artery disease; CT, computed tomography; EHR, electronic health record.

The 2013 ACC/AHA cholesterol guidelines define clinical ASCVD of coronary origin as a history of acute myocardial infarction, stable or unstable angina, or coronary revascularization presumed to be of atherosclerotic origin.

The authors propose that the findings from this initiative may generate ideas for specific actionable steps that may improve the accuracy of automated EHR data query for the purposes of quality performance assessment. The first step would be to revise diagnostic coding schemes to match clinical guideline diagnoses that call for specific guideline‐recommended treatments. The second step would be to engineer alerts into the EHR that notify providers that the patient has a diagnostic code eligible for assessment of quality performance, which should then prompt the provider to give the appropriate treatment, to revise the coded diagnosis, or to document the appropriate exemption in a discrete queryable field. Third, providers need to work with data analytic teams to ensure that specific detection algorithms are adjusted as appropriate in real time. Finally, we should likely discontinue integration of administrative billing data, which is known to introduce significant error, and instead rely primarily on clinical data derived from the EHRs (Table 4).

Our findings have limitations. We sampled only patient charts of CAD cases classified by EHR data query as statin performance failures; we did not sample charts of CAD patients who were classified as statin prescription successes, per EHR query, nor did we ask whether EHR data query had missed detecting patients with coronary ASCVD. It is possible that EHR data query also erroneously misclassified cases as satisfying the statin prescription metric, which could actually overestimate the actual rate of performance. For this reason, we are unable to provide a revised statin prescription performance rate for eligible patients with CAD. Another limitation is that we also only included patients with coronary ASCVD meeting a class I recommendation for statin therapy by current ACC/AHA guidelines.20 We did not include other populations for which statin therapy is recommended, including those with peripheral arterial disease, cerebrovascular disease, diabetes mellitus, familial hypercholesterolemia, total low‐density lipoprotein >190 mg/dL, and patients whose 10‐year risk for cardiovascular events exceeds 7.5%.20 However, the additional complexity of these parameters would likely only further undermine the accuracy of our EHR‐based identification of eligible patients being prescribed a statin. In addition, our initiative measured only rates of statin prescription (or documentation of statin exemption); we did not evaluate for patient adherence to statins. Finally, the patient population studied during our review may not reflect characteristics of the broader population of patients with CAD, and thus generalization of our results should be avoided. Each institution should conduct its own review to determine the applicability of our findings.

Conclusion

Although EHR data query may conceivably represent an efficient way to measure provider performance at a population level, this approach did not perform well at our institution compared with physician adjudication in the accurate identification of cases for which the provider failed to prescribe a statin to eligible patients with CAD. In fact, our findings suggest that our institution's EHR‐based data query was highly prone to errors when measuring provider performance, and dependence on such methods to justify payer reimbursements may not reflect actual quality delivered. We propose possible actionable steps to improve EHR data query accuracy for our institution; however, further research prospectively validating our recommendations is required before they can be considered evidence‐based.

Sources of Funding

This initiative was supported by internal funds at the Emory Clinic, Atlanta, GA.

Disclosures

None.

Figure S1. Diagnosis and procedure codes (ICD‐9, CPT, and SNOMED codes) used to identify patients with coronary artery disease during query of our electronic health record system. CPT indicates Current Procedural Terminology; ICD‐9, International Classification of Diseases, Ninth Revision; SNOMED, Systematized Nomenclature of Medicine.

Figure S2. Diagnosis and procedure codes (ICD‐9 and CPT codes) used for determining baseline characteristics of the study population. CPT indicates Current Procedural Terminology; ICD‐9, International Classification of Diseases, Ninth Revision.

Click here for additional data file.