Measuring Preventable Outcomes: Global Cardiovascular Risk (GCVR).

The National Committee for Quality Assurance (NCQA) piloted a new approach to quality measurement meant to reduce avoidable cardiac events and improve overall population health. In this pilot, we investigated whether a standardized technical specification could sufficiently define a process to reliably generate predicted outcome scores from heterogeneous electronic clinical data systems (ECDS) [1]. Patient data were electronically extracted from four health care organizations and processed by the Archimedes, Inc. Global Outcomes calculator, generating scores indicating future cardiovascular event probability for each provider's patient population. These Global Cardiovascular Risk (GCVR) scores represent the gap between current level of care achieved and optimal care for each clinician's patients, with a greater score indicating better performance. As GCVR requires more patient data than do traditional quality measures, we addressed feasibility and data completeness questions in order to understand the prospects of a wholly new quality concept. This pilot successfully produced GCVR scores for 2,251 clinicians, representing approximately 60 percent of the total patient population under study. To our knowledge, this is the first time predictive models have been proposed for quality measurement, and our pilot successfully demonstrated that a predicted outcome measure is feasible using electronic patient data. However, new specification standards are required before this approach is fully scalable to a national quality reporting program.

Introduction

Quality measurement has been a vital component in monitoring the U.S. health care system’s quality of care—assessing the effectiveness of quality improvement initiatives and serving as the basis for pay-for-performance programs and public accountability reports [2]. However, current quality measure strategies are limited as they usually (1) focus on processes and treatment goals instead of health outcomes, (2) have a tendency to address a single risk factor or biomarker when patients often have multiple conditions, (3) focus on population thresholds that may not be relevant to individual patient risk [5], and (4) fail to take advantage of opportunities to engage and motivate patients [3].

Atherosclerotic cardiovascular disease (ASCVD) is the leading cause of death in the United States [4], and expenditures continue to be higher for it than for any other diagnostic group, with combined direct and indirect costs estimated at $320 billion in 2011 alone [5]. A personalized approach to quality measurement that accounts for patient preference could be particularly advantageous in minimizing the overall burden of cardiovascular-related death and disability in the United States. To tackle a problem of this magnitude, we propose to shift the current quality measurement focus from population assessments of individual indicators such as smoking status, hypertension management, and A1C control to one of patient-centric assessment using a model that conveys the likelihood of future adverse events. These patient-driven predicted outcomes can help clinicians recognize the specific characteristics of their patients—optimizing treatment of existing ASCVD—while concurrently improving the overall cardiovascular health of their patients by supporting decision-making in both prevention- and risk reduction discussions. The quality measurement approach proposed here continuously supports population health and incentivizes health care teams to engage patients and individualize care management plans based on goal-oriented, probability-weighted outcomes.

While person-centric risk models are desirable, their application to quality measurement presents a number of challenges. First, quality measure specifications that rely on electronic clinical systems as a primary data source require standard extraction protocols referencing structured data elements within the clinical databases. Current risk prediction tools in the United States are yet to be adopted broadly because they rely on manual searches of nonstandardized and unstructured patient data requiring significant resources [6]. Second, any new quality approach must not add to the existing measurement burden experienced by individual clinicians [78]. Finally, most predictive modelling tools have been developed for specific patient populations or unique data sets, making effective quality assessment on a larger scale difficult and very expensive [9].

To meet these challenges, a patient-specific approach is needed that highlights those particular factors most relevant to a clinician’s treatment of ASCVD on a patient-by-patient basis [10]. Our measure of GCVR scores quantifies how well clinicians manage risk and health outcomes across their entire patient population with respect to individual clinical biomarkers and patient characteristics [11]. Linking contributing variables, derived from multiple sources, into a global, predicted outcome measure that accommodates the nuanced interactions between the ensemble of data points informing a person’s risk also provides relevant information about health care quality. The results from these calculations can be specified to assess effectiveness of care at many levels and could stimulate positive change in preventing avoidable events and lowering overall cardiovascular risk. The broader adoption of electronic clinical data systems and quality reporting standards makes it possible to systematically and consistently extract, transform and load (ETL) patient data needed for calculating GCVR, and reduces the burden of manual data extraction and measure calculation. Automating a structured process for clinical data query also introduces the possibility of returning quality information to clinicians in time to actually support decisionmaking. While much recent progress has been made in terms of electronic clinical data standardization and systems interoperability, many clinical practices still do not use standardized data, oftentimes documenting data within free-text fields that are not easily accessible for electronic-based quality measure reporting [6]. Therefore, a necessary first step in changing the quality model for cardiovascular care was systematically evaluating the consistency and completeness of structured electronic data. This paper reports on a pilot effort that investigated whether a standard technical specification could sufficiently define a process that would reliably generate outcome scores from heterogeneous data collected through ECDS.

Methods

NCQA recruited four sites to test the feasibility of generating clinician GCVR scores based on data from each organization’s electronic clinical systems. Participating organizations included one nonprofit health plan, one large nonprofit academic medical center, and two integrated health service organizations. Test sites were required to have a fully operational electronic health record system during the measurement period, the ability to warehouse data elements related to cardiovascular risk, and prior experience in reporting quality measure results. In order to obtain patient-level data from external organizations, NCQA first crafted a series of explicit instructions and technical requirements for reporting valid and reliable data. These specifications took the form of a GCVR Data Model (Appendix A) that describes the format of the necessary structured data elements, and a step-by-step process for submission through NCQA’s secure portal. A value set directory was also provided, specifically defining each individual data element in the measure using standard terminologies. This field test consisted of retrospective, electronic database research and a small number of key informant interviews assessing clinician views on the topic. A consent form, a semistructured protocol for our key informant interviews, and request for a waiver of Health Insurance Portability and Accountability Act (HIPAA) authorization on behalf of the participating organizations for this study—as allowed by 45 C.F.R. § 164.512(i) of the Privacy Regulations—were reviewed by the Institutional Review Board (IRB). As these data were not created prospectively for research purposes and were therefore determined not to be human subjects research, a waiver of HIPAA authorization for a limited data set was granted by the IRB.

At the start of the process, each component of the technical specification and protocol was first reviewed by independent subject matter experts and then by pertinent experts from each participating organization. Comments and feedback were incorporated as needed prior to going live with data collection. Once data collection was complete, all sites provided detailed feedback on their experience with the technical documentation as well as their personal experiences in programming the queries to extract the requested data elements from their local systems. In order for a new quality measurement approach to be considered feasible, all participating sites must have been able to follow the protocol as instructed and to generate the requested data in sufficient quantity and of sufficient quality for consistent generation of GCVR scores.

Our next step was to select a risk calculator. There are many risk calculator options available—each with its own set of data requirements, a unique algorithm for determining future outcomes, and a variable level of evidence supporting its risk estimations [1213141516]. For this research, we selected an evidence-based, predictive-risk calculator provided by Archimedes Inc. (http://www.sphanalytics.com/indigo/) that utilizes a large number of clinical variables to generate relevant risk predictions for the population under study [1718]. For each individual patient, the Archimedes calculator uses three scalable treatment-outcome scenarios, which are assembled to produce a GCVR score for each provider [19]. Scenario 1 starts by determining a “no care” risk profile by reversing out any current treatments, eliminating the potential benefits of current care in order to estimate risk as if the patient were not under any treatment. Scenario 2 determines a profile for each patient corresponding to actual current performance levels. Scenario 3 determines a “target” by which each patient will have achieved an optimal level of care. Each scenario has a corresponding rate of predicted outcomes. Using each patient’s data to determine these three points enables the GCVR score to reflect the proportion of preventable adverse events that are being prevented at current levels of care of the total number of preventable adverse events estimated under optimal care.

Each organization identified an eligible population of patients ages 18 to 85 years whose records included retrospective data collected during the 24-month period between January 1, 2011 and December 31, 2012 and produced a .csv data file populated with patient encounters, health risk assessments, and other electronically available information. One of the integrated delivery systems (IDS 1) and the academic medical center limited their eligibility to patients with a prior diagnosis of hypertension, diabetes, or cardiovascular disease (high-risk population), while the second integrated delivery system (IDS 2) and the health plan included any patient meeting the encounter timing and age criteria. This information was then sent to NCQA via a secure file transfer service for verification and for cataloguing of any of the data elements that were missing or misclassified. Next, the files were securely uploaded to the Archimedes server to generate GCVR scores (Figure 1 and 2). Finally NCQA matched GCVR scores to clinician National Provider Identifiers (NPIs) and organization IDs in the research database to analyze the calculated GCVR performance results.

Figure 1

Data Flow Process for Feasibility Testing

Figure 2

Clinician GCVR Scores by Test Site

Results

The four organizations successfully returned in excess of 480,000 individual patient records to NCQA for analysis. From this data set, GCVR scores were generated for 2,251 clinicians using 277,780 of those patient records. Records were excluded from the GCVR scoring process due either to data incompleteness for the period under study or to inaccurate data (i.e., not exactly matching the criteria as specified in the technical documentation). We intentionally did not enable any imputation functions available within the Archimedes calculator for this study as we wished to assess the proportion of raw data that could be produced by each of the organizations, following the provided specification. Table 1 presents the aggregated organizational results, illustrating the potential scale of the GCVR score for the total patient population. The (n) is the number of patients at each site with sufficient data to generate a GCVR score. Current Averted Events represents the predicted number of events that, under current levels of treatment, will be prevented in that patient sample, and Optimal Averted Events represent the total number of potentially avoidable events in the population if all patients were receiving optimal treatment based on their individual risk profiles. The GCVR score represents the variance between these two estimates for the sample population.

Table 1

Organizational GCVR Score by Test Site

	GCVR SCORE*	n	CURRENT AVERTED EVENTS	OPTIMAL AVERTED EVENTS
IDS 1**	69.7	45,198	1,418	2,036
IDS 2	76.5	145,277	1,420	1,856
AMC**	59.7	40,155	896	1,499
Health Plan	59.1	47,150	346	585

Notes: *A higher score is better. ** IDS = Integrated Delivery System; AMC = Academic Medical Center.

Table 2 presents the numeric details of the scatter plots for each organization. The (n) in Table 2 represents NPIs to which the clinician GCVR scores were attributed within each organization.

Table 2

Clinician GCVR Scores by Test Site

SITE	n	AVG	SD	PCT10	PCT25	PCT50	PCT75	PCT90
IDS 1	1083	70.3	11.2	58.0	66.0	72.2	77.3	81.0
IDS 2	24	76.0	5.4	74.2	75.2	76.4	77.9	78.3
AMC	1052	60.1	18.5	32.8	57.1	64.7	72.2	75.6
Health Plan	92	48.8	21.4	10.6	45.5	55.7	62.4	69.4

In testing the feasibility of GCVR, clinician workflow and decision-making about what is transcribed into the patient record was of foremost interest. It became evident that while it is relatively easy to execute electronic queries in many EHRs, altering clinical workflow patterns to accommodate new measure data requirements are difficult if members of the health care team do not immediately see how the change benefits the patient. For this reason, the GCVR team used a process of continuous feedback throughout the testing process as solutions to each of the various technological challenges were considered.

Discussion

Almost 60 percent of the patients in our system had sufficient structured data for us to calculate a GCVR score, demonstrating the feasibility of calculating predicted outcome scores using a large number of patient variables. Despite the increased use of national standards, longitudinal clinical data are still represented in many different formats.

Even relatively common clinical data elements are not always stored in standardized, structured fields; they contain repeating data elements across irregular time frames (e.g., outpatient blood pressure readings) and may include patient assessment information from several settings—posing a difficulty in constructing measures like the GCVR. Because a GCVR-based measurement framework requires more data than does typical performance reporting, our pilot was critical to understanding whether complex technical specifications could be efficiently implemented. Understanding how to communicate necessary data requirements such that they are uniformly interpreted by multiple organizations that can then report valid quality results is crucial in developing a new quality measure. Since our main focus was on feasibility, we needed to deliver a highly specific request in a common format to overcome the lack of standardization and to establish a process by which disparate data sets could be transformed (manually at first, but electronically in the future) to our GCVR data model that would then permit consistent and reliable calculation of outcome scores.

Now that feasibility has been successfully demonstrated, a measure like GCVR can become a cornerstone for a new quality framework that assesses the influence of modifying any one of a patient’s factors to see if it affects an individual’s likelihood of an adverse event. GCVR offers a unique opportunity to assess progress by relying on advanced technology to identify patients’ optimal treatment targets, encouraging clinical preventive strategies that set risk reduction goals, then measuring the attainment of those goal [2021]. As the complexity of electronic clinical data increases and variables indicative of treatment multiply measure developers must consider more efficient ways of assessing clinical care to produce timely and actionable information. An analytic model that accurately predicts cardiovascular outcomes in heterogeneous populations would be useful, despite the additional complexity because the new quality framework prioritizes a patient-centric approach. Our research findings demonstrate that well-defined technical specifications and a protocol requiring structured data can provide such solutions for quality measurement, regardless of site-specific limitations.

Limitations

The limited scope of this study prevented us from performing a multidimensional quality analysis of the data files, or an evaluation of the semantic and syntactic accuracy of the data extracted [22]. However, our success in producing GCVR scores from heterogeneous clinical data encompassing more than 270,000 patient records demonstrates the feasibility of the process from a data completeness perspective [23]. In the present study certain data types—such as medications and patient history—presented particular challenges that we worked around by requiring that separate files be submitted by participants, which could be individually validated and integrated into the patient-level file. As both measure developers and health care organizations obtain more experience with the level of technical specification required to ensure consistent ETL processes, we expect many of these steps to become unnecessary in future iterations.

Conclusion

To our knowledge, this is the first time that predictive models have been proposed for quality measurement. This pilot successfully demonstrated that a predicted outcome measure is feasible using electronic patient data. However, new specification standards are required before this approach is fully scalable to the level of a national quality reporting program. As experience with reporting measures using ECDS increases, improvements in quality reliability and standardization will follow–facilitating improved cardiovascular risk predictions and advancing new measurement concepts in health care quality assessment. GCVR’s transformative strategy offers a valuable opportunity to evaluate quality improvement by assessing patient-specific health outcomes on a national scale.

Table A1

NCQA GCVR Common Data Model

			.CSV VARIABLE INFORMATION
PRIORITY	DATA ELEMENT	DESCRIPTION	TYPE	FIELD LENGTH	VALUE	NULL VALUE
PATIENT DEMOGRAPHICS
Critical	RAND_ID	Patient unique identifier	Num	10	Any numeric ID unique to an individual patient	–
Critical	Org_ID	Field Test Organizational ID (assigned by NCQA)	Char	2	e.g., AA	–
Med	Provider_ID1	Provider identifier 1	Num	10	Any numeric ID unique to an individual provider	NI
Med	Provider_ID2	Provider identifier 2	Num	10	Any numeric ID unique to an individual provider	NI
Med	Payer_ID	NCQA Plan ID (if available)	Num	5	Numeric ID unique to the primary payer	NI
Med	SNAPSHOT_DATE	Date data was extracted (YYYY-MM-DD)	Date	10	YYYY-MM-DD	NI
High	DOB_YR	Patient DOB	Date	7	YYYY-MM	NI
Critical	SEX	Patient Gender	Char	1	Report the applicable code	–
low	ETHNICITY	Patient's ethnicity	Char	6	Report the applicable code present in the patient's record	NI
High	SMOKER	Tobacco User	Char	12	Report the applicable code present in the patient's record.	NI
Low	RACE	Patient's race	Char	6	Report the applicable code present in the patient's record	NI
MEDICAL HISTORY
High	DM	Diagnosis Diabetes (Type I or Type II)	Char	10	Report code with the period delimiter (if applicable: e.g., 493.10)	NI
High	D_HYP	Diagnosis Hypertension	Char	10	Report code with the period delimiter (if applicable: e.g., 493.10)	NI
High	D_IHD	Diagnosis Ischemic Heart Disease	Char	10	Report code with the period delimiter (if applicable: e.g., 493.10)	NI
High	D_ANGI	Diagnosis Angina	Char	10	Report code with the period delimiter (if applicable: e.g., 493.10)	NI
High	D_COATH	Diagnosis Coronary Atherosclerosis	Char	10	Report code with the period delimiter (if applicable: e.g., 493.10)	NI
High	D_CoAO	Diagnosis Coronary Artery Occlusion	Char	10	Report code with the period delimiter (if applicable: e.g., 493.10)	NI
High	D_CVD	Diagnosis Cardiovascular Disease	Char	10	Report code with the period delimiter (if applicable: e.g., 493.10)	NI
High	D_OPREA	Diagnosis Occlusion or Stenosis of Precerebral Arteries	Char	10	Report code with the period delimiter (if applicable: e.g., 493.10)	NI
High	D_ATH_RA	Diagnosis Atherosclerosis of Renal Artery	Char	10	Report code with the period delimiter (if applicable: e.g., 493.10)	NI
High	D_ATH_EXT	Diagnosis Atherosclerosis of Native Arteries of the Extremities	Char	10	Report code with the period delimiter (if applicable: e.g., 493.10)	NI
High	D_OCC_EST	Diagnosis Chronic Total Occlusion of the Artery of the Extremities	Char	10	Report code with the period delimiter (if applicable: e.g., 493.10)	NI
High	D_ART_THROM	Diagnosis Arterial Embolism and Thrombosis	Char	10	Report code with the period delimiter (if applicable: e.g., 493.10)	NI
High	D_ATH_EMB	Diagnosis Atheroembolism	Char	10	Report code with the period delimiter (if applicable: e.g., 493.10)	NI
High	PREVIOUS_MI	Previous Acute Myocardial Infarction	Char	10	Report code with the period delimiter (if applicable: e.g., 493.10)	NI
High	PREVIOUS_STROKE	Prior Stroke (not including Transient Ischemic Attack)	Char	10	Report code with the period delimiter (if applicable: e.g., 493.10)	NI
High	HF	Diagnosis Heart Failure	Char	10	Report code with the period delimiter (if applicable: e.g., 493.10)	NI
High	AF	Diagnosis Atrial Fibrillation	Char	10	Report code with the period delimiter (if applicable: e.g., 493.10)	NI
High	REVASC	Prior Revascularization (Coronary Artery Stent or Graft)	Char	10	Report code with the period delimiter (if applicable: e.g., 493.10)	NI
Med	LVH	Diagnosis Left-Ventricular Hypertrophy	Char	10	Report code with the period delimiter (if applicable: e.g., 493.10)	NI
N/A	GEST_DM	Gestational Diabetes	Char	10	Report code with the period delimiter (if applicable: e.g., 493.10)	NI
N/A	ESRD	End Stage Renal Disease (Stage IV or V CRF or ESRD)	Char	10	Report code with the period delimiter (if applicable: e.g., 493.10)	NI
N/A	PREG	Pregnancy	Char	10	Report code with the period delimiter (if applicable: e.g., 493.10)	NI
N/A	COGIMP	Cognitive Impairment	Char	10	Report code with the period delimiter (if applicable: e.g., 493.10)	NI
N/A	MAJ_DEP	Major Depression	Char	10	Report code with the period delimiter (if applicable: e.g., 493.10)	NI
EXAMINATION
High	WEIGHT	Weight Measurement (in pounds)	Num	3	Report numeric weight (in pounds) e.g., 178	NI
High	HEIGHT	Height Measurement (in inches)	Num	3	e.g., 072	NI
High	BP1_DATE	Date of Blood Pressure Measurment	Date	10	YYYY-MM-DD	NI
High	BP1_DIA	Diastolic Blood Pressure Reading (mmHg)	Num	3	e.g., 090	NI
High	BP1_SYS	Systolic Blood Pressure Reading (mmHg)	Num	3	e.g., 140	NI
Labs
High	A1C1_DATE	Date of HBA1C lab test	Date	10	YYYY-MM-DD	NI
High	A1C1_VAL	HBA1C lab test result (%)	Num	4	e.g., 06.5	NI
High	CHOL1_DATE	Date of Total Cholesterol Lab Test	Date	10	YYYY-MM-DD	NI
High	CHOL1_VAL	Total Cholesterol Result (mg/dL)	Num	3	e.g., 230	NI
High	HDL1_DATE	Date of HDL Lab Test	Date	10	YYYY-MM-DD	NI
High	HDL1_VAL	HDL Result (mg/dL)	Num	3	e.g., 055	NI
High	LDL1_DATE	Date of LDL	Date	10	YYYY-MM-DD	NI
High	LDL1_VAL	LDL Result (mg/dL)	Num	3	e.g., 101	NI
High	TRIG1_DATE	Date of Triglyceride	Date	10	YYYY-MM-DD	NI
High	TRIG1_VAL	Triglyceride Result (mg/dL)	Num	3	e.g., 101	NI
High	SERCR1_DATE	Date of Serum Creatinine	Date	10	YYYY-MM-DD	NI
High	SERCR1_VAL	Serum Creatinine (mg/dL)	Num	3	e.g., 1.5	NI
High	FPG1_DATE	Date of Fasting Plasma Glucose	Date	10	YYYY-MM-DD	NI
High	FPG1_VAL	Fasting Plasma Glucose Result (mg/dL)	Num	3	e.g., 101	NI
OTC MED STATUS
Med	ASPIRIN_STATUS	Taking aspirin	Char	10	Report the applicable code present in the patient's record	NI
Low	FISH_OIL_STATUS	Using fish oil or eating equivalent number of fish meals per week	Char	10	Report the applicable code present in the patient's record	NI
Low	NIACIN_STATUS	Taking crystalline niacin	Char	10	Report the applicable code present in the patient's record	NI
MEDICATION ALLERGY (OPTIONAL)
High	ASPIRIN_ALLERGY	Indicates patient is allergic to aspirin	Char	10	Report the applicable code present in the patient's record	NI
High	ACE_ALLERGY	Indicates patient is allergic to ACE Inhibitors	Char	10	Report the applicable code present in the patient's record	NI
High	ARB_ALLERGY	Indicates patient is allergic to ARBs	Char	10	Report the applicable code present in the patient's record	NI
High	BETA_ALLERGY	Indicates patient is allergic to Beta Blockers	Char	10	Report the applicable code present in the patient's record	NI
High	CCB_ALLERGY	Indicates patient is allergic to Calcium Channel Blockers	Char	10	Report the applicable code present in the patient's record	NI
High	DIURETIC_ALLERGY	Indicates patient is allergic to diuretics	Char	10	Report the applicable code present in the patient's record	NI
High	STATIN_ALLERGY	Indicates patient is allergic to statins	Char	10	Report the applicable code present in the patient's record	NI
MEDICATION HISTORY
High	DISPENSE_DT	Date of dispense	Date	10	YYYY-MM-DD	NI
Med	MED_CAT	Category of medication	Char	15	Antidiabetic, Insulin, ACE, ARB, Beta, CCB, Diuretic, Loop diuretic, Statin, Fibrate, Aspirin, Anticoagulant, Antiplatelet	NI
High	GPI/NDC/RxNorm	GPI, NDC, or RxNorm code	Num	14	Report the applicable code present in the patient's record	NI
High	DAYS_SUPPLY	Number of days supplied for current dispense	Num	3	e.g., 090, 230	NI
High	QUANTITY	Amount of medication in dispense.	Num	3	e.g., 090, 230	NI
High	SOURCE	Source of prescription information (EHR/Payer)	Num	2	01 = EHR02=Payer	NI