Angina Severity, Mortality, and Healthcare Utilization Among Veterans With Stable Angina.

Background Canadian Cardiovascular Society (CCS) angina severity classification is associated with mortality, myocardial infarction, and coronary revascularization in clinical trial and registry data. The objective of this study was to determine associations between CCS class and all-cause mortality and healthcare utilization, using natural language processing to extract CCS classifications from clinical notes. Methods and Results In this retrospective cohort study of veterans in the United States with stable angina from January 1, 2006, to December 31, 2013, natural language processing extracted CCS classifications. Veterans with a prior diagnosis of coronary artery disease were excluded. Outcomes included all-cause mortality (primary), all-cause and cardiovascular-specific hospitalizations, coronary revascularization, and 1-year healthcare costs. Of 299 577 veterans identified, 14 216 (4.7%) had ≥1 CCS classification extracted by natural language processing. The mean age was 66.6±9.8 years, 99% of participants were male, and 81% were white. During a median follow-up of 3.4 years, all-cause mortality rates were 4.58, 4.60, 6.22, and 6.83 per 100 person-years for CCS classes I, II, III, and IV, respectively. Multivariable adjusted hazard ratios for all-cause mortality comparing CCS II, III, and IV with those in class I were 1.05 (95% CI, 0.95-1.15), 1.33 (95% CI, 1.20-1.47), and 1.48 (95% CI, 1.25-1.76), respectively. The multivariable hazard ratio comparing CCS IV with CCS I was 1.20 (95% CI, 1.09-1.33) for all-cause hospitalization, 1.25 (95% CI, 0.96-1.64) for acute coronary syndrome hospitalizations, 1.00 (95% CI, 0.80-1.26) for heart failure hospitalizations, 1.05 (95% CI, 0.88-1.25) for atrial fibrillation hospitalizations, 1.92 (95% CI, 1.40-2.64) for percutaneous coronary intervention, and 2.51 (95% CI, 1.99-3.16) for coronary artery bypass grafting surgery. Conclusions Natural language processing-extracted CCS classification was positively associated with all-cause mortality and healthcare utilization, demonstrating the prognostic importance of anginal symptom assessment and documentation.

Clinical Perspective

What Is New?

Canadian Cardiovascular Society (CCS) anginal symptom class extracted from clinical notes within the electronic health record was positively associated with higher rates of mortality, cardiovascular hospitalizations, coronary revascularization, and overall healthcare costs.

What Are the Clinical Implications?

Assessing and documenting anginal symptom burden at the point of care using the CCS classification system has prognostic implications.

Our findings underscore the value of documenting CCS classification for risk stratification, shared decision‐making, and allocation of resources to patient populations that would benefit the most from revascularization interventions or other medical measures.

Introduction

Stable angina affects >10 million Americans and is the presenting symptom in approximately half of patients with coronary artery disease.1, 2, 3 Contemporary evidence‐based interventions for stable angina include aggressive lifestyle modifications, pharmacotherapy, and coronary revascularization to improve anginal symptoms and overall health status.2 An accurate, comprehensive assessment of symptom severity at the point of care is difficult to estimate but holds important implications for shared medical decision‐making between patient and clinician, who work together to determine the specific types and intensity of interventions for angina treatment based on symptomatology.

Stable angina severity can be assessed in clinical practice and research settings by the physician using grading measures such as the Canadian Cardiovascular Society (CCS) angina classification system. The CCS angina classification is a physician‐reported symptom severity scale used to assess and grade physical‐activity symptoms on 4 levels: class I indicates angina with strenuous exertion; class II indicates angina with walking >200 yards on flat surfaces, climbing stairs rapidly, or in cold or emotional situations; class III indicates angina with walking 100–200 yards on flat surfaces; and class IV indicates angina at rest or with any physical activity.2, 4 CCS angina classification has been associated with coronary revascularization, myocardial infarction, cognitive impairment, and mortality in clinical trials and prospective registries.5, 6, 7 However, these associations have not been replicated in large, population‐based cohorts using electronic health record data sets. This is predominantly because a patient's CCS class is documented as unstructured free text within clinic notes and thus is not easily extractable for research purposes. As such, data demonstrating the importance of CCS classification and outcomes in large cohorts of community‐treated angina patients is limited, particularly in patients with newly diagnosed stable angina.

We undertook this study with 2 objectives: (1) identify CCS documentation in clinical notes within a large, integrated health system using natural language processing (NLP) techniques and (2) determine the association between initial CCS classification and all‐cause mortality and healthcare utilization. We hypothesized that an NLP algorithm could accurately identify CCS classification within a large electronic health record system and that higher CCS classification would be associated with progressively higher all‐cause mortality and healthcare utilization, independent of other baseline characteristics.

Methods

Study Design and Population

We conducted a retrospective cohort study using the Veterans Health Administration (VHA) clinical and administrative databases in 2 phases. First, to develop the NLP tool, we identified veterans with at least 1 inpatient or outpatient encounter with an International Classification of Diseases, Ninth Revision, Clinical Modification (ICD‐9‐CM) code of 413.x (angina pectoris) between January 1, 1999, and December 31, 2006. Next, we used a combination of pharmacy fill data, Current Procedural Terminology, and ICD‐9‐CM codes to identify veterans with a diagnosis of stable angina using the protocol adapted from Phelps et al (Table S1).8 We used this cohort to develop and validate the CCS classification NLP tool (see “Natural Language Processing”). In the second phase, we employed this NLP tool, described in detail below, to identify documented CCS classifications in narrative, free‐text clinical notes (eg, medical notes from office visits). We further restricted the cohort to veterans with a new diagnosis of stable angina and at least 1 CCS classification documented in clinical notes between January 1, 2006, and December 31, 2013. The index date was the first date of documentation of CCS classification for each veteran. Veterans were followed from the index date until the occurrence of the event of interest, death, or December 31, 2013, whichever occurred first.

To capture the association of CCS class at initial diagnosis with the study outcomes, the analysis was restricted to veterans without evidence of a prior diagnosis of coronary artery disease or angina using the Phelps et al criteria in the previous 1 year from the time of the first CCS class documentation (ie, patients were recently asymptomatic, not newly symptomatic; Figure 1).8 This technique has been shown to be an effective method for identifying a cohort free of coronary artery disease using a 1‐year look‐back period.9 One year was chosen because we sought to include veterans who were relatively close to their diagnosis of coronary artery disease to avoid biasing the primary outcome of all‐cause mortality.

Figure 1

Application of inclusion and exclusion criteria relative to index date. indicates International Classification of Diseases, Ninth Revision.

The University of Utah institutional review board approved this study with a waiver of informed consent, and the Salt Lake City Veterans Affairs Health Care System Research and Development Office approved this study. The data that support the findings of this study are available from the corresponding author on reasonable request and with appropriate Veterans Administration institutional review board and data use agreement approvals.

Data Sources

The VHA databases serve as repositories for clinical, pharmacy, and administrative data from >140 VHA hospitals and 1200 outpatient clinics in all 50 states, the District of Columbia, and unincorporated territories (Guam, American Samoa, Puerto Rico, and the US Virgin Islands), representing the largest integrated healthcare delivery network in the United States. We obtained demographic, clinical, and healthcare utilization data from the Corporate Data Warehouse. For the primary outcome of all‐cause mortality, the Vital Status file was used. Pharmacy data were obtained from Managerial Cost Accounting, Pharmacy Benefits Management, and the Corporate Data Warehouse. Cost data were obtained from Managerial Cost Accounting. Notes from CART‐CL (VA Cardiovascular Assessment, Reporting and Tracking System for Cath Labs), a documentation system for cardiac catheterization procedures, were not included to avoid bias toward veterans undergoing revascularization procedures. Each veteran was assigned a unique deidentified number to link data sets and data tables.

Natural Language Processing

A rule‐based information extraction tool was developed to identify mentions of CCS classifications and to extract the values from free‐text clinical notes (Figure 2 and Table S2). The system was built using Leo, an NLP architecture developed by Veterans Informatics and Computing Infrastructure (VINCI) that uses a set of libraries that facilitate rapid creation and deployment of Apache UIMA‐AS (Unstructured Information Management Architecture Asynchronous Scaleout).10, 11 The knowledge base for the system was created using NLP‐assisted annotation based on a manual bootstrapping process. Precision validation was performed using the Chex Validation tool,12 and recall validation was performed using the eHOST application.13, 14 The system achieved 93.1% precision and 75.7% recall. See Data S1 for further detail.

Figure 2

Development of the natural language processing tool. CAD indicates coronary artery disease; , International Classification of Diseases, Ninth Revision; PPV, positive predictive value.

Outcome Measures

The primary outcome was all‐cause mortality. Secondary outcomes included all‐cause hospitalizations, cardiovascular‐related hospitalizations (ie, acute coronary syndrome, heart failure, and atrial fibrillation), coronary revascularization procedures (ie, percutaneous coronary intervention [PCI] and coronary artery bypass grafting surgery), and 1‐year direct healthcare costs (ie, inpatient, outpatient, and pharmacy costs). Cardiovascular‐related hospitalizations and revascularization procedures were determined using ICD‐9 and ICD‐10 codes. The coding algorithms and data sources used to identify the outcomes in the analysis are available in Table S3. Costs are presented in 2013 US dollars.

Covariates

Demographic variables were obtained from the Corporate Data Warehouse. Clinical characteristics and comorbidities were identified from Medical SAS data tables in VINCI using ICD‐9‐CM codes to define these covariates during the 1‐year pre–index date look‐back period (Table S3). We calculated a Charlson Comorbidity Index for each patient using algorithms developed by Quan et al that have been used to estimate prevalence of common comorbidities in administrative data and account for disease burden within outcomes research.15 We constructed medication profiles for each veteran using dispensing data for medications overlapping the index date. In addition, we ascertained baseline characteristics for veterans with and without a CCS classification available. For all multivariable‐adjusted regression analyses, covariates were chosen a priori and included in 2 nested models. Model 1 was adjusted for age, sex, and race/ethnicity. Model 2 included the variables in model 1 with additional adjustment for dyslipidemia, diabetes mellitus, hypertension, heart failure, smoking status, body mass index, and medication use (aspirin, statins, β‐blockers, calcium channel blockers, long‐acting nitrates, angiotensin‐converting enzyme inhibitors, and angiotensin receptor blockers) at baseline. The same covariates were used for cost adjustments as in models 1 and 2.

Statistical Analysis

Characteristics of the study population were calculated according to initial CCS classification (“baseline”). Trends in baseline characteristics across CCS classification were determined by modeling the characteristic of interest as the outcome variable and CCS classification as an ordinal predictor; appropriate regression models (ie, linear, logistic) were selected based on the type of variable (ie, continuous, binary) with dummy indicators created for categorical characteristics. We calculated incidence rates for each outcome as the number of events divided by the total person‐years at risk. Multivariable Cox proportional hazards regression models were used to calculate hazard ratios (HRs) for all‐cause mortality, hospitalizations, and revascularization procedures associated with CCS classification, with the lowest class (CCS class I) serving as the reference group. Potential violations of the proportional hazards assumption were tested by modeling the interaction between each model covariate and the log of follow‐up time. No violations were present. To compare healthcare costs at 1‐year follow‐up between groups, adjusted incremental costs were calculated using multivariate‐adjusted generalized linear models with a log‐link function and a γ distribution with cost as the outcome variable and CCS class as an ordinal predictor. Finally, we compared baseline characteristics between veterans with and without CCS class measurements as identified using the NLP tool (Table S4). Analyses were performed using SAS v9.2 (SAS Institute) and STATA v12.0 (StataCorp).

Results

Patient Characteristics

Of 299 577 veterans who met criteria for a diagnosis of stable angina, 14 216 veterans (4.7%) were newly diagnosed with at least 1 documented CCS classification. The distribution of CCS classes I, II, III, and IV at baseline was 28% (n=3983), 39% (n=5576), 28% (n=3978), and 5% (n=679), respectively (Figure 3). The mean age of the cohort was 67±9.8 years, 98% were men, and 81% were white (Table 1). Hypertension and hyperlipidemia were present in more than half the study population and did not differ between CCS classifications. Veterans with greater angina severity were more likely to have a diagnosis of diabetes mellitus, heart failure, and a higher mean Charlson comorbidity index score at baseline. Medical therapy did not vary significantly between CCS classifications at baseline except for long‐acting nitrates, which were more common among veterans with CCS classes II and III.

Figure 3

Flowchart of inclusion criteria defining study population. CAD indicates coronary artery disease; CCS, Canadian Cardiovascular Society (angina classification); , International Classification of Diseases, Ninth Revision; NLP, natural language processing.

Table 1

Baseline Characteristics by CCS Classification

Characteristic	CCS Classification				P Value for Trend
	I (n=3983)	II (n=5576)	III (n=3978)	IV (n=679)
Age, y	67±9.8	66±9.6	67±9.9	67±10.2	0.8159
Male sex	3927 (99)	5491 (98)	3910 (98)	668 (98)	0.3312
Race
White	3235 (81)	4469 (80)	3186 (80)	574 (85)	0.8453
Black	447 (11)	662 (12)	495 (12)	58 (9)	0.8600
Asian	74 (2)	102 (2)	50 (1)	14 (2)	0.1544
American Indian	28 (1)	61 (1)	41 (1)	5 (1)	0.3081
Unknown/missing	199 (5)	282 (5)	206 (5)	28 (4)	0.8220
Commercial insurance
Yes	2053 (52)	2926 (52)	2095 (53)	364 (54)	0.2261
No	1242 (31)	1729 (31)	1279 (32)	212 (31)	0.4459
BMI, kg/m2
Underweight	25 (1)	32 (1)	32 (1)	9 (1)	0.0651
Normal	649 (16)	802 (14)	602 (15)	105 (15)	0.2501
Overweight	1483 (37)	2019 (36)	1337 (34)	244 (36)	0.0040
Obese	1826 (46)	2723 (49)	2007 (50)	321 (47)	0.0010
Comorbidities
CCI	1.1±2.1	0.9±1.9	1.2±2.1	1.2±2.0	0.0278
Dyslipidemia	2476 (62)	3393 (61)	2376 (60)	425 (63)	0.1335
Diabetes mellitus	1743 (44)	2617 (47)	1955 (49)	335 (49)	<0.0001
Hypertension	3155 (79)	4475 (80)	3202 (80)	548 (81)	0.1443
Heart failure	770 (19)	894 (16)	875 (22)	138 (20)	0.0017
Smoking	819 (21)	1146 (21)	825 (21)	159 (23)	0.2981
Alcohol abuse	242 (6)	317 (6)	234 (6)	45 (7)	0.9261
Medications
β‐Blocker	2852 (72)	4049 (73)	2859 (72)	473 (70)	0.6501
Calcium channel blocker	934 (23)	1359 (24)	1015 (26)	159 (23)	0.1175
Long acting nitrate	1238 (31)	2230 (40)	1631 (41)	255 (38)	<0.0001
Aspirin	1362 (34)	1926 (35)	1420 (36)	221 (33)	0.5275
Statin	2857 (72)	4025 (72)	2845 (72)	469 (69)	0.3371
ACEI or ARB	2462 (62)	3408 (61)	2399 (60)	396 (58)	0.0562
P2Y12 inhibitor	878 (22)	1232 (22)	969 (24)	147 (22)	0.0699

Data are number of patients (%) or mean (SD). ACEI indicates angiotensin‐converting enzyme inhibitor; ARB, angiotensin II receptor blocker; BMI, body mass index; CCI, Charlson comorbidity index; CCS, Canadian Cardiovascular Society.

All‐Cause Mortality

During a median follow‐up of 3.4 years (interquartile range:1.6–5.6 years), all‐cause mortality rates were 4.58, 4.60, 6.22, and 6.84 per 100 person‐years for CCS classes I, II, III, and IV, respectively (Table 2 and Figure 4). The adjusted HRs for all‐cause mortality associated with classes II, III, and IV compared with class I were 1.05 (95% CI, 0.95–1.15), 1.33 (95% CI, 1.20–1.47), and 1.48 (95% CI, 1.25–1.76), respectively (P<0.001 for trend).

Table 2

Incidence Rates and HRs for Mortality and Hospitalizations by CCS Classification

Outcomes	CCS Classification				P Value for Trend
	I (n=3983)	II (n=5576)	III (n=3978)	IV (n=679)
All‐cause mortality
Incidence rate	4.59	4.60	6.22	6.84
HR (95% CI)
Unadjusted	1.00 (reference)	1.00 (0.90–1.10)	1.36 (1.22–1.50)	1.49 (1.25–1.76)	<0.0001
Model 1*	1.00 (reference)	1.04 (0.94–1.15)	1.40 (1.26–1.55)	1.55 (1.31–1.84)	<0.0001
Model 2†	1.00 (reference)	1.05 (0.95–1.15)	1.33 (1.20–1.47)	1.48 (1.25–1.76)	<0.0001
All‐cause hospitalization
Incidence rate	34.56	36.96	43.26	42.51
HR (95% CI)
Unadjusted	1.00 (reference)	1.09 (1.03–1.14)	1.21 (1.15–1.28)	1.21 (1.09–1.33)	<0.0001
Model 1*	1.00 (reference)	1.09 (1.04–1.14)	1.21 (1.15–1.28)	1.21 (1.09–1.33)	<0.0001
Model 2†	1.00 (reference)	1.10 (1.04–1.15)	1.19 (1.13–1.26)	1.20 (1.09–1.33)	<0.0001
Acute coronary syndrome
Incidence rate	2.24	2.35	2.56	2.87
HR (95% CI)
Unadjusted	1.00 (reference)	1.06 (0.92–1.23)	1.14 (0.98–1.33)	1.28 (0.98–1.67)	0.05
Model 1*	1.00 (reference)	1.08 (0.94–1.25)	1.15 (0.99–1.34)	1.31 (1.00–1.70)	0.04
Model 2†	1.00 (reference)	1.07 (0.93–1.24)	1.10 (0.95–1.29)	1.25 (0.96–1.64)	0.09
Heart failure
Incidence rate	4.08	3.66	4.17	3.97
HR (95% CI)
Unadjusted	1.00 (reference)	0.92 (0.82–1.03)	1.01 (0.90–1.14)	0.98 (0.78–1.23)	0.91
Model 1*	1.00 (reference)	0.95 (0.85–1.06)	1.03 (0.91–1.16)	1.00 (0.80–1.25)	0.81
Model 2†	1.00 (reference)	0.98 (0.88–1.10)	1.00 (0.88–1.12)	1.00 (0.80–1.26)	0.95
Atrial fibrillation
Incidence rate	6.83	6.09	7.67	7.06
HR (95% CI)
Unadjusted	1.00 (reference)	0.91 (0.84–1.00)	1.11 (1.01–1.22)	1.04 (0.88–1.25)	0.23
Model 1*	1.00 (reference)	0.93 (0.85–1.02)	1.12 (1.02–1.23)	1.07 (0.89–1.27)	0.17
Model 2†	1.00 (reference)	1.00 (0.92–1.09)	1.06 (0.96–1.16)	1.05 (0.88–1.25)	0.46

Incidence rates are n per 100 person‐years or risk ratio (95% CI) unless noted otherwise. CCS indicates Canadian Cardiovascular Society; HR, hazard ratio.

Model 1 includes adjustment for age, sex, and race/ethnicity.

Model 2 includes variables in model 1 and additional adjustment for dyslipidemia, diabetes mellitus, hypertension, heart failure, smoking status, body mass index, and medication use at baseline (statins, β‐blockers, long‐acting nitrates, calcium channel blockers, aspirin, angiotensin‐converting enzyme inhibitors, angiotensin receptor blockers).

Figure 4

Cumulative hazard of all‐cause mortality by CCS class. The median follow‐up time is 3.4 years (interquartile range: 1.6–5.6 years). CCS indicates Canadian Cardiovascular Society (angina classification).

Hospitalizations

The incidence of all‐cause hospitalizations significantly increased with higher CCS class (P<0.0001 for trend; Table 2 and Figure 5). After full multivariable adjustment, the adjusted HRs for experiencing at least 1 all‐cause hospitalization associated with classes II, III, and IV compared with class I were 1.10 (95% CI, 1.04–1.15), 1.19 (95% CI, 1.13–1.26), and 1.20 (95% CI, 1.09–1.33), respectively (P<0.0001 for trend). There was no association between CCS class and hospitalizations for atrial fibrillation or heart failure. Hospitalizations for acute coronary syndrome increased with CCS class, although this trend was significant only in the unadjusted model (unadjusted P=0.05 for trend; fully adjusted P=0.09 for trend).

Figure 5

Cumulative hazard of all‐cause and cause‐specific hospitalizations by CCS class. A, All‐cause hospitalizations. B, Hospitalizations due to acute coronary syndrome. C, Hospitalizations due to atrial fibrillation. D, Hospitalizations due to heart failure. CCS indicates Canadian Cardiovascular Society (angina classification).

Revascularizations

A significant association was noted between increasing CCS class and risk of revascularization by either PCI or coronary artery bypass grafting (Table 3 and Figure 6). The fully adjusted HRs for PCI procedures for classes II, III, and IV relative to class I were 1.51 (95% CI, 1.25–1.83), 1.87 (95% CI, 1.53–2.27), and 1.92 (95% CI, 1.40–2.64), respectively (P<0.0001 for trend). The fully adjusted HRs for coronary artery bypass grafting for classes II, III, and IV relative to class I were 1.72 (95% CI, 1.48–1.99), 2.26 (95% CI, 1.94–2.62), and 2.51 (95% CI, 1.99–3.16), respectively (P<0.0001 for trend).

Table 3

Incidence Rates and HRs for Revascularization Procedures by CCS Classification

Outcomes	CCS Classification				P Value For Trend
	I (n=3983)	II (n=5576)	III (n=3978)	IV (n=679)
PCI
Incidence rate	1.13	1.64	2.12	2.22
HR (95% CI)
Unadjusted	1.00 (reference)	1.50 (1.24–1.81)	1.83 (1.51–2.23)	1.94 (1.41–2.67)	<0.0001
Model 1*	1.00 (reference)	1.48 (1.22–1.79)	1.82 (1.50–2.21)	1.91 (1.39–2.63)	<0.0001
Model 2†	1.00 (reference)	1.51 (1.25–1.83)	1.87 (1.53–2.27)	1.92 (1.40–2.64)	<0.0001
CABG
Incidence rate	1.87	3.20	4.40	4.93
HR (95% CI)
Unadjusted	1.00 (reference)	1.75 (1.51–2.03)	2.20 (1.90–2.55)	2.52 (2.01–3.18)	<0.0001
Model 1*	1.00 (reference)	1.73 (1.49–2.00)	2.20 (1.89–2.55)	2.49 (1.98–3.14)	<0.0001
Model 2†	1.00 (reference)	1.72 (1.48–1.99)	2.26 (1.94–2.62)	2.51 (1.99–3.16)	<0.0001

Incidence rates are n per 100 person‐years or risk ratio (95% CI) unless noted otherwise. CABG indicates coronary artery bypass grafting; CCS, Canadian Cardiovascular Society; HR, hazard ratio; PCI, percutaneous coronary intervention.

Model 1 includes adjustment for age, sex, and race/ethnicity.

Model 2 includes variables in model 1 and additional adjustment for dyslipidemia, diabetes mellitus, hypertension, heart failure, smoking status, body mass index, and medication use at baseline (statins, β‐blockers, long‐acting nitrates, calcium channel blockers, aspirin, angiotensin‐converting enzyme inhibitors, angiotensin receptor blockers).

Figure 6

Cumulative hazard of revascularizations. A, Percutaneous coronary intervention (PCI) procedures. B, Coronary artery bypass grafting (CABG) procedures. CCS indicates Canadian Cardiovascular Society (angina classification).

Costs

The mean total healthcare costs in the 1‐year follow‐up period were $28 674, $29 347, $35 416, and $34 771 among those with CCS classes I, II, III and IV, respectively (Table 4). The fully adjusted, total incremental costs associated with classes II, III, and IV, relative to class I, were $1184 (95% CI, −$876 to $3244), $6160 (95% CI, $3720–8600), and $6006 (95% CI, $1239–10 773), respectively (P for trend=0.0009). Mean outpatient healthcare costs among CCS classes I, II, III, and IV were $12 934, $12 856, $14 387, and $14 257, respectively. Mean inpatient healthcare costs among CCS classes I, II, III, and IV were $13 010, $13 932, $18 113, and $17 842, respectively. Fully adjusted pharmacy costs were similar between CCS class groups (P=0.98 for trend).

Table 4

One‐Year Total and Incremental Healthcare Costs by CCS Classification

Costs*	CCS Classification				P Value for Trend
	I (n=3983)	II (n=5576)	III (n=3978)	IV (n=679)
Total (all settings)
Cost for 1 y, mean (median)	28 674 (13 187)	29 347 (14 229)	35 416 (17 448)	34 771 (17 771)
Incremental cost (95% CI)
Unadjusted	1.00 (reference)	663 (−1653 to 2978)	6759 (3973–9544)	6111 (675–11 548)	0.002
Model 1*	1.00 (reference)	671 (−1600 to 2942)	6443 (3724–9161)	6386 (1020–11 751)	0.001
Model 2†	1.00 (reference)	1184 (−876 to 3244)	6160 (3720–8600)	6006 (1239–10 773)	0.0009
Outpatient
Cost for 1 y, mean (median)	12 934 (8302)	12 856 (8411)	14 387 (9424)	14 257 (9579)
Incremental cost (95% CI)
Unadjusted	1.00 (reference)	−83 (−679 to 513)	1458 (775–2141)	1328 (27–2629)	0.005
Model 1*	1.00 (reference)	−129 (−725 to 467)	1390 (708–2072)	1323 (21–2625)	0.005
Model 2†	1.00 (reference)	168 (−422 to 757)	1426 (756–2096)	1319 (49–2590)	0.006
Inpatient
Cost for 1 y, mean (median)	13 010 (0)	13 932 (0)	18 113 (0)	17 842 (0)
Incremental cost (95% CI)
Unadjusted	1.00 (reference)	918 (−1130 to 2965)	5114 (2510–7718)	4842 (−427 to 10 110)	0.01
Model 1*	1.00 (reference)	1095 (−831 to 3022)	4962 (2534–7389)	5379 (308–10 450)	0.004
Model 2†	1.00 (reference)	1299 (−468 to 3066)	4742 (2544–6940)	5050 (531–9570)	0.003
Pharmacy
Cost for 1 y, mean (median)	2730 (1661)	2559 (1635)	2916 (1894)	2671 (1867)
Incremental cost (95% CI)
Unadjusted	1.00 (reference)	−172 (−363 to 19)	187 (−32 to 406)	−58 (−443 to 328)	0.76
Model 1*	1.00 (reference)	−164 (−347 to 19)	179 (−31 to 389)	−62 (−430 to 307)	0.79
Model 2†	1.00 (reference)	−174 (−365 to 17)	91 (−124 to 307)	−83 (−463 to 297)	0.98

All costs are in 2013 dollars. CCS indicates Canadian Cardiovascular Society.

Model 1 includes adjustment for age, sex, and race/ethnicity.

Model 2 includes variables in model 1 and additional adjustment for dyslipidemia, diabetes mellitus, hypertension, heart failure, smoking status, body mass index, and medication use at baseline (statins, β‐blockers, long‐acting nitrates, calcium channel blockers, aspirin, angiotensin‐converting enzyme inhibitors, angiotensin receptor blockers).

Discussion

In this retrospective cohort analysis of veterans with stable angina, documented CCS classification in the electronic health record identified by NLP was positively associated with all‐cause mortality and healthcare utilization for cardiovascular causes. Compared with CCS class I, those with CCS class III or IV angina symptom severity near the time of angina diagnosis experienced 33% to 48% higher all‐cause mortality rate over median 3‐year follow‐up. All‐cause mortality rates were similar among those presenting with class I and II symptoms. A graded association was also noted in which CCS class was associated with progressively higher risk of undergoing coronary revascularization (ie, PCI and coronary artery bypass grafting procedures). CCS classes III and IV were also significantly associated with more incurred direct healthcare costs. To our knowledge, the current analysis represents the first large, electronic health record–based cohort study reporting CCS classification and its association with all‐cause mortality in newly diagnosed stable angina patients. These results support the prognostic importance of assessing and documenting angina severity at the point of care.

To date, several studies have shown mixed results regarding the association between angina severity and clinical outcomes, but these studies have generally been limited to hospitalized patients undergoing revascularization.5, 7, 16, 17, 18 Clinical and survey data from ACQUIP (Veterans Affairs Medical Center Ambulatory Care Quality Improvement Project) demonstrated that greater angina severity scores, as measured by the Seattle Angina Questionnaire (SAQ), were associated with all‐cause mortality and increased admission rates for acute coronary syndrome.17, 18 Conversely, a post hoc analysis from the BARI 2D (Bypass Angioplasty Revascularization Investigation 2 Diabetes) registry found similar rates of all‐cause death and myocardial infarction among patients with CCS classes III and IV compared with those with CCS classes I and II.19 Nonetheless, in light of the lack of consensus regarding the most appropriate method for assessing and documenting anginal symptoms at the point of care (ie, physician assessment with the CCS or patient‐reported using the SAQ), objective evidence of myocardial ischemia (eg, inducible ischemia using exercise treadmill testing with stress echocardiography), as opposed to angina symptoms alone, is a stronger marker of adverse cardiovascular risk.19, 20 Although our study did not investigate CCS classification with objective evidence of myocardial ischemia on noninvasive testing, our findings justify the utility of CCS classification and documentation in routine practice. The relative merits, utility, and value of provider‐documented symptom severity (eg, the CCS) compared with patient‐reported symptom assessments (eg, the SAQ) remain to be fully understood.21 Programs collecting patient‐reported outcomes as part of routine clinical care, such as the University of Utah's mEVAL initiative, may provide insights into the relative value of provider‐ versus patient‐reported outcome measures in contemporary practice.22

Our study found that veterans with higher CCS class were more likely to undergo revascularization procedures after CCS documentation; unexpectedly, however, we did not observe more aggressive medication therapy in those with more severe angina symptoms at baseline. The COURAGE (Clinical Outcomes Utilizing Revascularization and Aggressive Drug Evaluation) trial did not demonstrate a mortality benefit with revascularization via PCI compared with medical therapy alone for patients with stable angina.23 Similarly, ORBITA (Objective Randomised Blinded Investigation with optimal medical Therapy of Angioplasty in stable angina) demonstrated no statistically significant difference in exercise time between patients with severe (≥70%) single‐vessel coronary artery stenosis who underwent a PCI procedure compared with a sham procedure.24 In context of the COURAGE and ORBITA results, our findings highlight an area for improving the extent to which medical therapy is optimized before attempting revascularization, especially in light of our finding of higher healthcare utilization and costs among those with more severe anginal symptoms (ie, CCS III and IV patients); however, additional research is needed on best practice strategies to ensure medical optimization before PCI.

Our analyses demonstrate that inpatient and healthcare costs are higher among veterans who present with more severe anginal symptoms. In contrast to other studies assessing angina‐associated healthcare costs, the greatest proportion of total healthcare costs observed were incurred during inpatient care, followed by outpatient and pharmacy costs.25 Notably, pharmacy costs did not differ significantly across CCS classes; this result may reflect greater use of revascularization strategies over medical therapy in patients with more debilitating angina.

Our study has several strengths, including the large sample size and use of NLP for extraction of CCS classifications from free‐text medical notes from a large integrated healthcare database. The use of NLP in research settings to identify exposures is increasing and has been used in recent years to identify critical limb ischemia and bleeding based on clinical notes,26, 27 to predict mortality based on intensive care nursing notes,28 to recognize pneumonia among chest radiograph reports,29 to estimate medication dosing,30, 31, 32 and more. The VHA is the largest integrated delivery network in the United States and thus offers the advantage of capturing comprehensive inpatient and outpatient healthcare encounters, laboratory, procedural, billing, and pharmacy data. Our study has several limitations worth noting. The use of CCS classification itself has a number of weaknesses because it is ultimately a physician‐based assessment rather than a patient‐reported outcome, such as the SAQ.19, 33 In addition, in our study, only a small proportion of the entire incident angina population had a CCS classification recorded (≈5%) despite the high precision of the NLP tool (93%). This finding introduces the possibility of an inherent bias in the documentation of CCS class toward sicker patients who may be more likely to undergo revascularization procedures. The recall of our tool, although high at 75%, also introduces potential bias because ≈25% of documented CCS cases could not be identified with our NLP tool and thus could not be included in the analysis. When comparing patients with and without CCS classification available, a larger percentage of patients without CCS classification had documentation by a cardiologist. In addition, common comorbidities (eg, diabetes mellitus and hypertension) were more prevalent in those with a CCS classification than in those without, and Charlson comorbidity index was higher (indicating more comorbidity burden) in those without a CCS classification. Consequently, those veterans included in our study may not be representative of the larger angina population. Furthermore, the external validity of our findings is limited by the use of a single healthcare system with a nearly exclusively male and largely white population. Our study cannot address the important differences in angina symptoms based on race or sex, which have been described.17 Given that we used VHA administrative data for utilization outcomes, we did not capture utilization that occurred outside the VHA. Finally, as with all observational studies, although we adjusted for important confounders when assessing outcomes, unmeasured variables may affect our results.

Conclusion

Our study found that greater angina severity at the time of angina diagnosis, as measured by CCS classification identified from clinical notes using NLP techniques, was associated with higher rates of mortality, cardiovascular hospitalizations, coronary revascularization, and overall healthcare costs. Furthermore, these outcomes were largely graded, with stronger associations present as severity increased, and persisted after controlling for variables known to affect differences in angina symptoms, prognosis, and treatment.17 These findings are of particular relevance to clinicians’ efforts to judiciously utilize revascularization interventions, to improve patients’ symptomatic burden from angina, and to reduce overall healthcare costs. The results of the current analysis support the prognostic importance of assessing and documenting angina severity at the point of care.

Sources of Funding

This work was supported using resources and facilities at the Veterans Affairs (VA) Salt Lake City Health Care System with funding from VA Informatics and Computing Infrastructure VA HSR HIR 08‐204 and the University of Utah and by an investigator‐initiated grant from Gilead Sciences.

Disclosures

Drs Bress, LaFleur, and Nelson and Mr Crook received support from the Investigator Sponsored Research Grant from Gilead Sciences. Drs Dodson and King currently serve as consultants for Novartis Pharmaceuticals on a research project unrelated to the current study. Dr Dodson is supported by a Patient Oriented Career Development Award (K23AG052463) from the National Institute of Health/National Institute of Aging and a Mentored Clinical and Population Research Award from the American Heart Association. Dr Shah is supported by a grant from the National Heart, Lung, and Blood Institute (K08HL136850). The remaining authors have no disclosures to report.

Data S1. Supplemental Methods.

Table S1. Phelps et al's Criteria for Classification of Chronic Stable Angina

Table S2. Codes Used to Define Variables in the Analysis

Table S3. Characteristics of Angina Patients With and Without a Canadian Cardiovascular Society Class Documentation

Table S4. Examples of Used Patterns and Phrases That Match the Patterns

Click here for additional data file.