Applied Use of Composite Quality Measures for EHR-enabled Practices.

INTRODUCTION: The Primary Care Information Project (PCIP) of the New York City Department of Health and Mental Hygiene has been assisting providers to implement health information technology such as electronic health records (EHRs) since its founding in 2005. Currently, all practices affiliated with PCIP are offered technical support services in order to improve the use of the EHR. We studied the performance of clinical practices on EHR-derived Composite Quality Measures (CQMs) over time. Because specific EHR functionalities are important to calculating the quality measures, we hypothesize that performance on each of the CQMs will differ according to the EHR functionalities, and that this can inform the process of developing targeted technical assistance for the practices.
METHODS: We created four CQMs: (1) Screening, (2) Assessment, (3) Control-BP, and (4) Control-Other. Using data from 93 practices, we identified three tertiles of CQM performance (premier, average, and low tiers) for each measure. A scatterplot of CQMs in 2010 versus 2011 was used to examine the individual movement of practices by tier. A dependent t-test compared the change in mean CQMs, and a chi-square test examined the association between the score and performance tier changes.
RESULTS: Over a one-year period, low tier practices demonstrated the highest gains, average tier practices had modest gains, and premier tier practices had gains in some measures, but losses in others. On the Screening CQM 70 percent of practices remained within the same tier, with 60 percent on Assessment, 52 percent on Control-BP, and 38 percent on Control-Other; the Control-Other group showed the greatest improvement. DISCUSSION: By considering EHR functionalities associated with each of the four CQMs, we suggest that technical assistance can be better targeted to low-tier performing practices. In addition, there is still the potential for improvement over time at practices more familiar with key functionalities.

Introduction

The New York City Department of Health and Mental Hygiene’s Primary Care Information Project (PCIP) assists clinical practices with the implementation of electronic health records (EHRs). PCIP collaborates with EHR-enabled practices and monitors population health by estimating health care quality measures on aggregate patient information from over 700 practices across New York City.1,2

Beyond supporting the process of care, the EHR is a data repository that combines clinical information reported by both providers (clinical notes, diagnoses, etc.) and patients (immunization, adherence).3 The full potential in EHR-based quality measurement is the ability to benchmark quality measures that accurately reflect the processes of care, are clinically relevant, and are trusted by all stakeholders.4 However, implementations of EHRs do not automatically improve quality of care. Steps have to be taken to ensure data quality by confirming clinical accuracy and completeness of data elements in the right format (mostly structured-data fields) to support estimation of the quality measures.5

PCIP provides technical assistance to clinical practices located in medically underserved areas across New York City. Ongoing technical assistance is important to realize the full potential of the EHR-based quality measure estimation, particularly for small primary care practices that are more likely to be impacted by the obstacles to EHR implementation—such as financial and technical barriers, and concerns about productivity loss.6 The technical assistance is provided by clinical quality specialists on site and remotely, with the frequency of visits determined by either PCIP staff or practice requests. Recent findings from our efforts have shown that high intensity technical assistance (more than eight visits) can significantly improve performance on quality measures.7 Yet, sustaining technical assistance can be costly; the challenge for PCIP and similar organizations lies in determining what technical assistance to provide in order to improve and sustain performance on quality measures.

In a prior analysis, we created 4 composite measures from 13 individual clinical measures that depict clinical snapshots of care in areas of prevention and chronic disease management.8 The 4 CQMs were (1) Screening, (2) Assessment, (3) Control-BP, and (4) Control-Other. The composites and corresponding quality measures are shown in Table 1. The grouping of the individual measures by factor analysis was reflective of the clinical care and also shared EHR functionalities, similar to prior studies that linked higher performance on quality measures with EHR functionalities.9

Table 1.

Relevant EHR Functionalities Associated with the Composite Measures

COMPOSITE QUALITY MEASURES (CQM)	QUALITY MEASURES	DEMOGRAPHICS	VITALS	PROBLEM LIST/ASSESSMENTS1	LABORATORY2	SMART FORM3	DRUGS
Screening	HIV screening	DOB		HIV	HIV
	A1c testing	DOB		DM	A1c
	LDL testing (high risk)	DOB		DMIVD	LDL
	Cholesterol screening (general population)	DOB, gender		no DMno IVD	HDL, total
Control-Other	Cholesterol control (general population)	DOB, gender		no DMno IVD	HDL, LDL, total cholesterol
	A1c control (< 7%)	DOB		DM	A1c
	LDL control (high risk)	DOB		DMIVD	LDL
Control-BP	BP control in IVD (140/90)	DOB	Most recent BP	IVDno DM
	BP control in HTN (140/90)	DOB	Most recent BP	HTNno DM or IVD
	BP control in DM (130/80)	DOB	Most recent BP	DM
Assessment	Antithrombic tx (IVD or DM)	DOB		DMIVD			Antithrombic
	Smoking cessation intervention	DOB		Smoking		Smoking statusFax to quit	Smoking cessation
	Asthma symptom assessment	DOB		Asthma		Asthma

Notes:

Problem list or Assessments contain disease diagnosis

Laboratory refers to these components: LOINC codes, values in yellow boxes, reviewed radio button, received checkbox, received date—all filled in.

A standard questionnaire to aid assessment of conditions

Table Legend

Date of Birth

Ischemic Vascular Disease

Diabetes Mellitus

Low density Lipoprotein

Hypertension

Human Immunodeficiency Virus

The Hemoglobin A1c (HbA1c)

High Density Lipoprotein

From the list of individual measures, we identified a group of control measures that shared clinical similarities, but which were factored into two groups because of the shared EHR functionalities: the Control Other measures (control of low-density lipoprotein, cholesterol, and A1c) required laboratory test results, whereas the Control-BP measures (control of blood pressure among hypertensive patients, patients with ischemic heart disease, and patients with diabetes) required the entry of systolic and diastolic pressure in the vitals section. The Screening measures (HIV testing, cholesterol screening, and hemoglobin A1c testing) required minimal interaction with the EHR, and the Assessment measures (smoking cessation intervention, asthma symptoms assessment, and antithrombotic therapy) also required less interaction but were dependent on data captured in a standard questionnaire (smart form) and information recorded as drug therapy.8

By studying the performance of clinical practices on CQMs over one year, it is apparent which CQMs may require more technical assistance than others to support practices. Because specific EHR functionalities are important to calculating the quality measures, we hypothesize that performance on each of the CQMs will differ according to the EHR functionalities and that this can inform the process of developing targeted technical assistance for the practices.

Methods

There are 700 small clinical practices in New York City transmitting data to PCIP. To follow the performance of the clinical practice over a one-year period from 2010 to 2011, we included clinical practices that were EHR enabled at least a year before the baseline period (2010), and were consistently transmitting all the adult related measures used in the estimation of the CQMs a year prior to the baseline. All practices have collaborated with the New York City Department of Health and Mental Hygiene during the EHR implementation process; technical assistance has been ongoing, before and after EHR implementation. Besides typical technical assistance of EHR implementation, PCIP provides help with Meaning Use of EHR, PCMH recognition and pay-for-performance programs such as the Health eHearts program.27

To assess practice improvement in light of other factors that may have an impact on CQMs, we created three tertiles to compare performance: premier, average, and low tiers. These tiers were based on the distributions of each CQM from our previous study. The tertile cutoff points were used to create the three performance groups (premier, average, and low tiers) for each CQM. The lowest tertile was classified as the low tier, the middle tertile as the average tier, and the highest tertile as the premier tier. A scatter plot of CQMs in 2010 versus 2011 was used to examine the individual movement of practices within each tier. Then a Pearson’s correlation was used to assess the relationship between the 2010 and 2011 scores, while a dependent t-test compared the change in mean CQMs, and a chi-square test examined the association between the score and performance tier changes. To follow the performance of the clinical practice over the one-year period from 2010 to 2011, we included clinical practices that were EHR enabled at least a year before the baseline period (2010) and that were consistently transmitting all the adult related measures used in the estimation of the CQMs a year prior to the baseline.

To evaluate the change in tiers, we categorized the changes from a higher tier to a lower tier as a decrease and from the lower tier to a higher tier as an increase. A practice was labeled “retained” if it remained within its initial tier. For each of the four composite scores, we used McNemar’s test to assess the statistical significance of practice movement across tiers from 2010 to 2011.

Results

Practice Characteristics

A subset of 93 practices met the criteria as stated in the method section to be included in this analysis. Table 2 presents the practice characteristics. At the end of the reporting period in 2011, mean time since EHR implementation was 35 months (std=7.16) and mean number of clinicians per practice was 2.33 (std=2.8) with 97 percent of practices having only one or two clinicians.

Table 2.

Practice Characteristics, 2011

n=93	MEAN (std)	RANGE
Months using EHR	35.00 (7.16)	22–51
Providers	2.33 (2.8)	1–16
	NUMBER (%)	NUMBER (%)
Sites	Single Site76 (82%)	Multisite17 (18%)
Organization	Small Practice90 (97%)	Community Health Center3 (3%)
Providers	Single Provider53 (57%)	Multiple Providers40 (43%)

Change in Performance Tiers from 2010 to 2011

Table 3 presents the changes in mean performance across tiers by composite score from 2010 to 2011. For the low and average tiers, we observed significant increases (P=<0.05) in practice performance on all four CQMs and also for the premier tier of the Screening CQM. The low tier consistently showed the greatest improvement with mean change ranging from 46.06 on the Control–Other score to 8.42 on the Screening score. The mean changes for the average tier ranged from 20.31 on the Control-Other score to 1.9 on the Control-BP score. The premier tier showed a significant decrease in mean score for Control-BP of 5.04.

Table 3.

Changes in Mean Performance Across Tiers by Composite Score, 2010–2011

COMPOSITE	SCORE	N	2010 MEAN (std)	2011 MEAN (std)	2011–2010 CHANGE (std)
Assessment	Low	31	22.82 (5.07)	33.22 (17.36)	10.40 (12.29)	*
	Average	30	35.43 (4.18)	40.23 (9.37)	4.80 (5.19)	*
	Premier	32	60.58 (15.04)	63.25 (18.53)	2.67 (3.49)
Control-BP	Low	30	32.92 (14.50)	49.20 (14.70)	16.28 (0.20)	*
	Average	31	56.11 (4.21)	58.01 (16.06)	1.90 (11.85)	*
	Premier	32	72.60 (7.24)	67.56 (12.89)	−5.04 (5.65)	*
Control-Other	Low	44	0.00 (0.00)	46.06 (23.19)	46.06 (23.19)	*
	Average	17	27.77 (9.61)	48.08 (23.53)	20.31 (13.92)	*
	Premier	32	65.84 (10.64)	60.07 (20.14)	−5.77 (9.50)
Screening	Low	30	21.94 (10.15)	30.36 (15.94)	8.42 (5.79)	*
	Average	31	46.52 (4.43)	52.04 (8.17)	5.52 (3.74)	*
	Premier	32	62.49 (5.97)	66.25 (10.37)	3.76 (4.40)	*

P≤0.05

Individual Practice Performance Gains

Figure 1 compares the performance of practices in 2010 to those in 2011.The rate of change, represented by the slope from years 2010 to 2011, is greatest for the Screening score (0.93), followed by Assessment (0.80), and then by both Control scores (0.25).When the 2010 scores were used to predict the 2011 score, as measured by Pearson’s correlation, a similar order was generated: Screening (0.89), Assessment (0.75), Control-BP (0.52), and Control-Other measures (0.33) (data not shown).

Figure 1.

Performance of Individual Practices Within Tiers: 2010 versus 2011, by Composite Score

As shown by a tighter cluster of practices on the scatter plot, the average tier had the lowest standard deviations for both years. The performance change on the Control–Other score stands out from the other three CQMs because a majority of practices in the low tier started off with a zero score but improved dramatically by 2011 (also see Table 3). Although the premier group maintained their high score, they showed wider variation across all the CQMs.

Movement Across Performance Tier Groups by CQM from 2010 to 2011

Figure 2 presents the changes in performance tiers on CQMs from 2010 to 2011. Overall, the association between the composite score and the performance tier change was significant (p=0.0009). On the Screening CQM 70 percent of practices remained within the same tier, with 60 percent on Assessment, 52 percent on Control-BP, and 38 percent on Control-Other. Conversely, the Control-Other group showed the greatest improvement. The McNemar’s tests were not significant—indicating that most practices remained within the same tier from 2010 to 2011.

Figure 2.

CQM Performance Tiers, from 2010 to 2011

Discussion

Our findings indicate that performance generally improved over time on all CQMs, and improvements differed within each composite score for the performance tiers. The general trend in improvement for the CQMs is similar to previous findings on the performance of the PCIP individual quality measures that showed improvement over time.10 A survey of the PCIP providers showed that as providers adjust to the EHR over time they are better able to use their EHR meaningfully,11 which accounts for the continuous improvement on the individual quality measures.10

The difference in performance across the CQMs can be attributed to shared EHR functionalities. Because the individual measures for each CQM share EHR functionalities as shown in Table 1, poor performance on a CQM helps to pinpoint problems with the relevant EHR functionalities neccessary to estimate that CQM. We observe a consistent improvement in scores across all tiers of the Screening and Assesment CQMs because these measures require mininal interaction with the EHR. The relatively high performance gains associated with the Screening and Control-Other CQMs could be due to the activation of the laboratory interface, which occurs only after the EHR has been implemented. The integration of laboratory results into the EHR is a complicated process that can have an impact on reporting of results and, subsequently, quality measure estimation.12,13 The only statistically significant decrease in any of the groups occurred in the premier tier of Control-BP measure, which is most likely due to clinical factors beyond the scope of this analysis, such as the severity of patients.

The difference between performance tiers could be attributed to possible barriers that are unique to each performance tier. This is because most of the practices remain within the same tier over time, and so the inability to improve relative to the other practices on each CQM could be due to other factors in addition to issues with EHR functionalities. Therefore, practices in the low tier may have adopted the EHR because of the incentivized Meaningful Use initiative,14 but they are still susceptible to factors that inhibit EHR implementation. EHR factors affecting implementation are likely to be the top three factors identified by users: perceived usefulness, productivity, and lack of motivation. Ease of use was found to be the strongest motivator for the EHR user.15–17 Those in the average tier, based on their clustering (Figure 1) and relatively high performance at baseline in comparison to the low tier, have overcome the barriers to implementation, but could be challenged by lack of knowledge or hindered by the perception of time burden associated with entry of structured data.18 Practices in the premier tier are conversant with functionalities necessary to achieve high scores, but sustained a decrease in scores because of the nature of BP treatment. A practice can treat easier patients and improve overall, but the difficult remaining patients will increase the denominator of BP without changing the numerator. Thus, there will be some negative movements of the control measure.

The performance differences across CQMs, together with the distinct changes within tiers, are the key to designing targeted technical assistance. The PCIP has reduced financial and technical barriers to implementation by subsidizing EHR licenses, selecting the EHR software vendors, and providing ongoing technical assistance before and after EHR implementation.7,19 However, some barriers are outstanding and are having an impact on practice performance on the composite measures. Low tier practices will benefit from comprehensive training with specific focus on addressing motivation and productivity.

Teaching users to further secure personal health information can improve their confidence in electronic records, thereby supporting the transition from paper records.. This can improve productivity and possibly reduce possible losses in revenue.17 Technical support to the average tier could aim at informing the users of the relevance of the quality measures and assisting them in adjusting their workflow and incorporating necessary functions into their daily routine. Incentives could help improve performance for all tiers but particularly for the premier tier; it can help sustain performance since incentives have been linked to relatively high performance on quality measures.20 In addition, a study of PCIP providers demonstrated that incentivized providers were more likely to show a greater interest in their performance.21,26

Besides technical support and assistance with workflow adjustment, collaboration between EHR users and vendors can influence EHR redesign22 to support the needs of the clinical providers while concurrently improving data capture without increasing the time burden on the user. The quality measures rely heavily on structured data, while clinical providers are trained in and prefer narrative that is supported by free text entry. Despite advantages such as data completeness, the structured data format takes more time and may require the use of additional interfaces, which means multiple steps, whereas the narrative method provides comprehensive information and maintains the medical relevance of the notes, as well as facilitating communication between providers.23–25

A compromise using a semistructured approach to data entry and capture can address provider preference and improve data quality. A proposed format is the structured narrative that uses real-time natural language processing (NLP) to encode text into Extensible Markup Language (XML) that transforms entered text into a format which is both human readable and machine readable. The coded entry is then matched against standardized coding schemes such as the Unified Medical Language System (UMLS), International Classification of Diseases 10 (ICD-10) or SNOMED Clinical Terms (CT). The success of this approach will be enabled by highly interoperable EHR systems.23

The time burden associated with assessing multiple interfaces16 can be eliminated by consolidating interfaces.24 Especially in the case of key chronic-disease management, a summary of the patient information on a single interface can guide providers in comprehensive management and facilitate ease of use. Vendors can be motivated to redesign the EHRs to support provider needs while capturing data for quality measure estimation by there being an impact on their revenue. The Office of the National Coordinator for Health Information Technology (ONC) certifies a list of EHRs that meet Meaningful Use requirements, and this drives demand for the certified EHRs. With revenue at stake, vendors are more likely to meet certification criteria set out by the ONC.

Conclusion

The objective of this analysis was to use EHR functionalities associated with composite quality measures to inform how implementing targeted technical assistance could help improve a practice (the targeted technical assistance provided at each tier was determined by a previous PCIP study27). Our findings indicate that practices are capable of quality improvement and that targeted technical assistance can further improve performance by addressing and overcoming specific barriers to EHR implementation and use. Further studies will focus on designing a targeted technical-assistance intervention for participating providers, and then evaluating the impact of the intervention over time while taking into account factors such as practice and provider characteristics that are not accounted for in this analysis.

As providers strive to attain the Meaningful Use stages, performance on EHR-based quality measures and relevant technical assistance will gain prominence. This analysis presents a viable approach to improving performance on EHR-based quality measures, particularly for small primary care practices in urban settings. Beyond the relevance of the findings to understanding the potential for improvement and Meaningful Use of quality measures, our findings can help to inform practices that quality measures during initial phases of EHR implementation may not always accurately reflect the process of care. A time lapse during which providers and practices address obstacles to EHR implementation will likely be necessary for many small practices before the EHR-based quality data can be used to draw valid conclusions about the quality of care.