Challenges to electronic clinical quality measurement using third-party platforms in primary care practices: the healthy hearts in the heartland experience.

Third-party platforms have emerged to support small primary care practices for calculating and reporting electronic clinical quality measures (eCQM) for federal programs like The Medicare Access and CHIP Reauthorization Act of 2015 (MACRA) and Merit-based Incentive Payment System (MIPS). Yet little is known about the capabilities and limitations of electronic health record systems (EHRs) to enable data access for these programs. We connected 116 small- to medium-sized practices with seven different EHRs to popHealth, an open-source eCQM platform. We identified the prevalence of following problems with eCQM data for data extraction in seven different EHRs: (1) Lack of coded data in five of seven; (2) Incorrectly categorized data in four of seven; (3) Isosemantic data (data within the incorrect context) in four of seven; (4) Coding that could not be directly evaluated in six of seven; (5) Errors in date assignment and labeled as historical values in five of seven; and (6) Inadequate data to assign the correct code in two of seven. We recommend specific enhancements to EHR systems that can promote effective eCQM implementation and reporting to MACRA and MIPS.

BACKGROUND

Since the passage of the 2009 Health Information Technology for Economic and Clinical Health Act and implementation of the electronic health records (EHRs) Incentive Programs, EHR adoption has rapidly climbed to nearly 100% for hospitals and 80% for ambulatory practices. However, widespread adoption has not led to widespread improvements in quality, efficiency, and outcomes, which may be in part attributable to the challenge with reliably measuring quality of care.

Numerous studies have described the challenges practices face in calculating and reporting electronic clinical quality measures (eCQMs). These include lack of structured data, low rates of data completeness, and poor data accuracy. Small primary care practices face additional challenges in eCQM reporting for myriad reasons, including cost to license additional EHR components to manage eCQMs, limited access to informatics expertise, and limitations of EHR systems., These challenges and others hinder the ability of small practices, who tend to have fewer health information technology resources the larger practices and health systems, to accurately calculate and report eCQMs. They also have financial implications under the Merit-based Incentive Payment System (MIPS), the federal quality reporting program passed as part of the Medicare Access and CHIP Reauthorization Act (MACRA).

As part of the Meaningful Use Certification process, EHR vendors have increasingly adopted various standards and implementation specifications designed for information exchange, quality measurement, and reporting., For EHR systems with limited eCQM capabilities, standardization potentially enables third-party platforms to consume exported data, calculate eCQMs, and report to the federal government and back to the clinical practice for quality improvement. The Centers for Medicare and Medicaid Services (CMS) certifies several vendors—including private companies and professional societies—for calculating and reporting eCQMs as registries. The growth in third-party platforms reflects an emerging opportunity to support practices in calculating and reporting eCQMs.

The capabilities of third-party platforms to perform eCQM calculations for small practices is contingent on a reliable method of extracting and analyzing EHR data through mechanisms such as standardized EHR data exports, direct database extracts, or open application programing interfaces (APIs). Little is known about the capabilities and limitations of EHRs to enable data access via standard or other mechanisms for third-party platforms in small- to medium-sized, primary care practices.

This work was conducted as part of EvidenceNOW, a national cooperative that tests strategies to improve performance of four cardiovascular quality measures: aspirin dosing, blood pressure control, cholesterol management, and smoking cessation. As part of that effort, our group connected practices’ EHRs to an open-source eCQM platform called popHealth, which has been certified by the Office of the National Coordinator for Health IT (ONC). During this process, we identified challenges to using third-party platforms for the calculation of four quality measures that are part of a national effort to improve cardiovascular prevention and care called the Million Hearts® initiative (Table 1).

Table 1.

Quality measures evaluated as part of the EvidenceNOW initiative

Measure	National quality forum number	Measure narrative	Coding systems
Aspirin when appropriate	0068	Percentage of patients 18 years of age and older who were discharged alive for acute myocardial infarction, coronary artery bypass graft, or percutaneous coronary interventions in the 12 months prior to the measurement period, or who had an active diagnosis of ischemic vascular disease during the measurement period, and who had documentation of use of aspirin or another antithrombotic during the measurement period.	HCPCS, SNOMED CT, CPT, ICD9CM, ICD10CM
Blood pressure control	0018	Percentage of patients 18–85 years of age who had a diagnosis of hypertension and whose blood pressure was adequately controlled (<140/90 mmHg) during the measurement period.	HCPCS, SNOMED CT, CPT, ICD9CM, ICD10CM, LOINC
Cholesterol management	n/a	Percentage of high-risk adult patients aged >= 21 years who were previously diagnosed with or currently have an active diagnosis of clinical atherosclerotic cardiovascular disease; OR adult patients aged >=21 years with a fasting or direct LDL-C level >= 190 mg/dL; OR patients aged 40–75 years with a diagnosis of diabetes with a fasting or direct LDL-C level of 70–189 mg/dL; who were prescribed or are already on statin medication therapy during the measurement year.	HCPCS, SNOMED, CPT
Smoking cessation	0028	Percentage of patients aged 18 years and older who were screened for tobacco use one or more times within 24 months AND who received cessation counseling intervention if identified as a tobacco user.	HCPCS, SNOMED CT, CPT, RXNORM

LDL-C: low-density lipoprotein cholesterol; HCPCS: The Healthcare Common Procedure Coding; System; SNOMED CT: Systematized Nomenclature of Medicine -- Clinical Terms; CPT: Current Procedural Terminology; ICD9CM: The International Classification of Diseases, Ninth Revision, Clinical Modification; ICD10CM: The International Classification of Diseases, Tenth Revision, Clinical Modification; LOINC: Logical Observation Identifiers Names and Codes.

METHODS

Sample

Healthy Hearts in the Heartland (H3) is a quality improvement cooperative dedicated to measuring and improving cardiovascular care in small- to medium-sized primary care practices within three states: Illinois, Indiana, and Wisconsin. From November 2015 to October 2016, we enrolled 226 small- to medium-sized practices with an average of 3.4 clinicians per practice. We defined small- to medium-sized practices as those with up to 25 primary care clinicians. Those practices utilized a total of 22 different EHR systems, seven of which we developed connectors to popHealth.

All practices who had one of the seven systems for which we developed connectors were offered to be connected to popHealth. We provided access to a customized version of popHealth for interested practices.

Software—popHealth

Starting with the ONC-certified v4.0.0 release of popHealth (https://github.com/OSEHRA/popHealth/tree/v4.0.0), we extended the system with user interface enhancements specific to our study needs, system enhancements to speed up measure calculations, and API extensions to allow automated data loading and running of calculations.

Data extraction strategies

Given the differences in EHR capabilities, we applied four strategies to export patient-level data from the EHR system in each practice. We applied the strategies in the following order, stopping when we identified a strategy that provided the most complete data set:

Direct database extracts—acquire a direct connection to the underlying database system the EHR runs on, and extract the specific fields and tables needed for measure calculations.

APIs—a system-level call (typically as a web service) that provided access to patient-level data.

Standard export—receive an export of all patient records in a standards-based format, including Consolidated-Clinical Document Architecture (C-CDA) or Continuity of Care Document (CCD),, which contain administrative, demographic, and clinical data in a universal format. This required the EHR to export all records in a single batch file for retrieval.

EHR-based reporting—for EHRs that provide additional reporting capabilities, we generated reports containing patient-level data. This included reports built into the EHR, as well as custom-developed reports if existing reports were not sufficient.

Once we found a successful data export strategy, we then attempted to automate the export or use a manual process to run and download extracts. Using proprietary software (Park Street Solutions; Naperville, IL), we securely uploaded data extracts from practice EHRs to a centralized data server. For each EHR, we established a base extract-transform-load pipeline and customized it as needed for individual practice EHRs. In addition to extracting data from the relevant fields, we also performed the following steps as needed:

Converted medication names and descriptions into RxNorm codes;

Converted smoking status names and descriptions into the appropriate measure Systematized Nomenclature of Medicine – Clinical Terms (SNOMED CT) value set object identifiers;

Split blood pressure results into systolic and diastolic components.

Analysis

During the implementation process, we evaluated the various strategies to access the clinical data necessary for each of the four quality measures of interest. The four quality measures required several data types including problem lists, encounter diagnostic codes, medication codes, smoking history, blood pressure measurements, and laboratory results (specifically, cholesterol measurements). For each EHR, we identified the potential options for retrieving patient-level data mentioned above and recorded how each vendor licensed the output mechanisms. We also examined whether we were able to batch individual-level data for output. Batching, which consists of extracting groups of individual-level data at one time in contrast to individually extracting each record, is essential to high-throughput quality measure calculation via third-party platforms. We assessed technical feasibility by review of available EHR documentation, individual evaluation of EHR capabilities, as well as discussions with vendor technical support when possible.

During the implementation process, the technical team (ANK, LVR, PK, and AS) systematically recorded each obstacle encountered during quality measure calculation as appropriate and developed solutions to those when technically feasible. The technical team then presented its categorization schema to the larger study team, which vetted, suggested edits, and ultimately approved a final schema. Two members of the technical team (LVR and AS) reviewed practice-level data and quantified the prevalence of each challenge at the practice-level and by EHR system.

RESULTS

Out of a total of 226 practices, a total of 153 had an EHR system for which we had the technical capabilities to develop popHealth connectors. Of the 153 practices eligible for popHealth connection, a total of 125 practices (81.7%) agreed to be connected to popHealth. Of those who agreed to be connected, we successfully connected a total of 116 practices (92.8%) with seven different EHR systems (Table 2) to popHealth.

Table 2.

Characteristics and capabilities of EHR systems connected to popHealth for third-party quality measurement

EHR vendor	Number of practices using EHR (total practices = 116)	Able to batch C-CDA/CCD output (Yes/No)	Selected method of data extractiona
A	8	Yes	Direct database extract
B	16	Yes	EHR-based reporting
C	13	Yes	Standard export (C-CDA)
D	1	Yes	Standard export (CCD)
E	71	Yes	Direct database extract
F	5	Yes	Direct database extract
Gb	2	Yes	Standard export (CCD)

C-CDA: Consolidated-Clinical Document Architecture; CCD: Continuity of Care Document; EHR: electronic health record.

Method of data extracted selected based on data quality and data completeness.

Was able to perform a data extract for baseline data, but unable to successfully export data afterward.

The primary reasons for failure to connect were: (1) the lack of a functional, licensed feature in the practice EHR system to generate reports; (2) Inability to batch CCD/C-CDA extracts; (3) Unable to access data from a third-party host; (4) and the practice subsequently declined to allow the connection to be established. We note that we were able to establish a connection to one of the EHR vendors (Vendor G in Table 2) to collect baseline data; however, subsequent attempts to refresh the data failed. This was due to lengthy processing times to export the CCD documents, which made the EHR unusable while running, and could not be scheduled for off-hours. None of the sites that agreed to be connected had implemented any of the remaining 15 EHR systems. Table 3 illustrates the practice characteristics for the practices stratified by connection status. Practice characteristics were overall similar except that a higher percentage of practices that were not connected to PopHealth (unable to be connected or declined) were clinician-owned solo or group practices. Of the 64 clinician-owned solo or group practices, 31 (48.4%) were not connected to popHealth due to either technical limitations (n = 9) or declined (n = 22).

Table 3.

Practice characteristics stratified by popHealth connection status

Variable	Connected to popHealth N = 116		Unable to extract data N = 9		Declined popHealth N = 28
Practice size (Mean + SD)
Clinicians	4.9 + 11.9		3.3 + 3.9		3.5 + 9.5
Clinical staff	7.3 + 20.1		7.0 + 9.0		6.0 + 15.3
Office staff	6.7 + 29.5		3.1 + 4.1		7.1 + 23.1
	N	%	N	%	N	%
Location
Urban	114	98.3	8	88.9	25	89.3
Rural	2	1.7	1	11.1	3	10.7
Practice ownership
Clinician-owned solo or group practice	33	28.5	9	100.0	22	78.6
Hospital/health system owned	43	37.0	0	0.0	4	14.3
Federally qualified health center or look-alike	38	32.8	0	0.0	2	7.1
Other	2	1.7	0	0.0	0	0.0
Meaningful use certified EHR system	114	98.3	7	77.8	26	92.9
Patient-centered medical home	37	31.9	1	11.1	3	10.7
Medically underserved area/population	46	39.7	3	33.3	4	14.3
Patients with public insurancea
High number of Medicare patients	49	42.2	2	22.2	10	35.7
High number of Medicaid patients	40	34.5	7	77.8	15	53.6
Specialty mix
Multi-specialty	35	30.2	0	0.00	7	25.0
Single specialty	61	52.6	8	88.9	19	67.9
Received additional revenue for efficient resource utilization	73	62.9	2	22.2	10	35.7
Works with a network or organization for eCQM reporting	95	84.8	11	73.3	12	44.4

High number was defined as above average for all practices who reported this total.

We identified and organized challenges into six categories of barriers to extracting eCQM data from our sample of EHRs: (1) Lack of coded eCQM data; (2) Incorrectly categorized eCQM data; (3) Isosemantic data; (4) Coding that could not be directly evaluated; (5) Errors in date assignment and labeled as historical values; and (6) Inadequate data to assign correct code (Table 4). Challenges were related to the implementation of the EHR vendor system, the lack of specificity of the standards, the measure specifications and associated value sets, and data entry by user. In terms of eCQM calculation, we identified data-related challenges that affected all three aspects of measure calculation: the numerator and denominator; inclusion; and exclusion criteria. “Inadequate data to assign correct codes” was present in two of seven EHR systems, while the other five were present in four to six systems. Within each practice, if a challenge were present, then it usually affected the majority of measures related to eCQM calculation and would likely affect other measures attempting to use the same type of data.

Table 4.

Challenges encountered during quality measure calculation using a third-party platform

Challenge	Description	Prevalence	Examples	Relevant stakeholders
Lack of coded data	Data in source EHRs frequently do not include codes from standard vocabularies. Data exported into standards can be unusable for quality measure calculation.	Seen in five of seven EHR systems	Tobacco use is often recorded as simple text as part of “assessment” data and recorded as a string or local code by EHR. In C-CDA documents code and value elements for medications and laboratory values frequently are ‘nullFlavor = “UNK”’ attribute likely due to absence of data in the source system or errors during export process.	User Vendor Standards
Incorrectly categorized data	Source systems organize data in terms of underlying coding systems, rather than in terms of the nature of the data itself.	Seen in four of seven EHR systems	One source system classified all CPT-4 codes as procedures and did not separately account where CPT-4 codes are used to describe the nature of encounters. Most EHR systems distinguish procedures from encounters.	Vendor
Isosemantic data	Data are available, coded, and accurate but presented in the wrong context	Seen in four of seven EHR systems	Blood pressure values are generally presented as “Observations,” but the measures expect to find blood pressure values as aspects of a procedure. Social history values for tobacco usage without an accompanying “Procedure” entry for the assessment of social history	Vendor Measure/Value set stewards
Coding that cannot be directly evaluated	Measure value sets for medications only contain RxNorm Codes for generics.	Seen in six of seven EHR systems	If EHR records anti-thrombotic medications as RxNorm code for trade name, then it would not be counted in the numerator of the aspirin measure.	Vendor Measure/Value set stewards
Errors in date assignment and labeled as historical values	Lack of dates assigned to problem lists, medications, social history, and diagnostic codes. Only most recent smoking status or blood pressure measurement is available in the export.	Seen in five of seven EHR systems	If aspirin is recorded as a historical medication without a start or end date, then it will not be included in the aspirin measure numerator. Inaccurate measure calculation due to using only most recent blood pressure or smoking status value.	User Vendor Standard Measure/Value set stewards
Inadequate data to assign correct codes	Data in source systems and exports lack sufficient detail to assign the correct code.	Seen in two of seven EHR systems	Laboratory results, such as LDL cholesterol value, are generally recorded with descriptions that lack sufficient detail to allow the choice of the correct LOINC code.	Vendor

DISCUSSION

In this report, we highlight challenges when using third-party platforms for extracting data from EHRs for eCQM measurement in small- to medium-sized primary care practices. The ability to use third-party platforms is particularly important among these practices because they may utilize EHRs with less functionality and have access to fewer resources for quality measurement and reporting.

The challenges range from connecting with EHR systems themselves to navigating the different data output mechanisms. These output mechanisms each have their own strengths and limitations and can vary by vendor, and at times, within the same vendor but at different practices. We identified common challenges with the EHR data, the EHR systems, the export mechanisms, the measures, and the value sets. Some of these challenges, such as problems with standards (C-CDA and CCD), have been previously described., However, to our knowledge, this is one of the first reports on the technical limitations of using third-party platforms for calculating and reporting eCQMs in the era after Meaningful Use Stages 1 and 2, as well as the challenges across the spectrum of quality measurement faced when using a third-party platform.

Although the EHR adoption rate among ambulatory practice has risen to 80%, end-user satisfaction remains mixed for many reasons including poor usability, increased time per patient visit after EHR adoption, lack of clinician trust in EHR data quality and measures, and lack of functionalities to facilitate accurate billing, adherence to clinical guidelines, and care coordination.,, Third-party platforms have the potential to improve quality of care and end-user satisfaction by facilitating quality reporting, which can be a challenging and resource-intensive task, especially for small practices. However, the effectiveness of these systems will be limited by difficulties accessing and analyzing data required for quality measurement.

As described elsewhere, some challenges may stem from structured data capture mechanisms by the end-user during patient clinic visits, but challenges we describe can occur even when the appropriate data are discretely captured at the point of care. These challenges span each stage of quality measurement: data entry at the time of patient visit, data storage in the vendor database, connecting to the vendor system, extracting data in bulk from the system, and running the measure engine on extracted data.

In light of the complexity of these challenges that span across multiple steps in eCQMs, a parsimonious solution is unlikely. Redesigning EHR systems and clinical workflows and developing natural language processing pipelines to better capture and store structured data using standards are essential., Developing eCQMs whose data elements reflect the actual data being captured, stored, and extracted in primary care practices may improve the ability to run eCQMs. The standards themselves (such as C-CDA) would benefit from additional modifications to facilitate quality measurement. While all EHRs in our sample were capable of generating C-CDAs or CCDs, we did not use this method for data access in most cases due to incomplete or missing historical data needed for quality measures. We note that the HL7 Fast Healthcare Interoperability Resources standard more adequately supports quality measurement, but its success will require EHR vendors to provide a robust implementation if it will improve upon existing C-CDA and CCD offerings. Furthermore, revising the EHR federal certification process by ONC to improve data capture, storage, and export essential to eCQM may stimulate changes to current EHR vendor systems and standards.

To achieve our goal of using a third-party platform for quality measurement, we had to customize popHealth and develop strategies for data extraction and transformation that varied by EHR system and, at time, at the practice level. This underscores the challenges in developing a single platform that can be used across different EHR systems and different types of practices. Moreover, our analysis highlights how the capabilities to use third-party platforms in clinician-owned solo and group practices may differ from health system-owned clinics or federally qualified health centers. Even with an experienced technical team, we were only able to connect less than half of all eligible clinician-owned practices for either technical reasons or because the practice declined the connection.

The main limitation of this report is that these findings represent the experience of Healthy Hearts in the Heartland Collaborative working with small- to medium-sized practices in the Midwest using a customized, third-party platform and may not be applicable to all EHR systems, practices, or geographic areas. Another limitation is that although we attempted to accurately evaluate the export capabilities for each vendor, we recognize that we may have not have exhaustively identified each potential method for data export given limited access to documentation, systems, and support. However, although other methods to export data may exist within the vendor systems, our approach represents the capabilities of vendor systems in real-world primary care practices identified by a team of experts.

In summary, we have highlighted the challenges of using third-party platforms for quality measurement in small- to medium-sized primary care practices. Clinical workflow and EHR system redesign and wider adoption of improved standards will be essential to improving the ability of primary care practices to receive accurate MACRA-MIPS financial payments tied to quality reporting.

FUNDING

This work was supported by the Agency for Healthcare Research and Quality 5R18HS023921. Dr. Ahmad was in part supported by the National Heart, Lung, and Blood Institute of the National Institutes of Health under Award T32HL069771 and the Agency for Healthcare Research and Quality under Award K12HS0323011. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality and the National Heart, Lung, and Blood Institute.

CONTRIBUTORSHIP STATEMENT

This manuscript was initiated and conceptualized by FA, LR, SP, AS, and AK All author contributed to the acquisition, analysis, or interpretation of data. FA and LR drafted the manuscript. All authors critically revised the manuscript and approved the final version.

COMPETING INTERESTS

Andy Schiever was the Chief Technology Officer of Park Street Solutions at the time of conducting this study.