Recalibration of the ACC/AHA Risk Score in Two Population-Based German Cohorts.

BACKGROUND: The 2013 ACC/AHA guidelines introduced an algorithm for risk assessment of atherosclerotic cardiovascular disease (ASCVD) within 10 years. In Germany, risk assessment with the ESC SCORE is limited to cardiovascular mortality. Applicability of the novel ACC/AHA risk score to the German population has not yet been assessed. We therefore sought to recalibrate and evaluate the ACC/AHA risk score in two German cohorts and to compare it to the ESC SCORE.
METHODS: We studied 5,238 participants from the KORA surveys S3 (1994-1995) and S4 (1999-2001) and 4,208 subjects from the Heinz Nixdorf Recall (HNR) Study (2000-2003). There were 383 (7.3%) and 271 (6.4%) first non-fatal or fatal ASCVD events within 10 years in KORA and in HNR, respectively. Risk scores were evaluated in terms of calibration and discrimination performance.
RESULTS: The original ACC/AHA risk score overestimated 10-year ASCVD rates by 37% in KORA and 66% in HNR. After recalibration, miscalibration diminished to 8% underestimation in KORA and 12% overestimation in HNR. Discrimination performance of the ACC/AHA risk score was not affected by the recalibration (KORA: C = 0.78, HNR: C = 0.74). The ESC SCORE overestimated by 5% in KORA and by 85% in HNR. The corresponding C-statistic was 0.82 in KORA and 0.76 in HNR.
CONCLUSIONS: The recalibrated ACC/AHA risk score showed strongly improved calibration compared to the original ACC/AHA risk score. Predicting only cardiovascular mortality, discrimination performance of the commonly used ESC SCORE remained somewhat superior to the ACC/AHA risk score. Nevertheless, the recalibrated ACC/AHA risk score may provide a meaningful tool for estimating 10-year risk of fatal and non-fatal cardiovascular disease in Germany.

Introduction

In 2013, the American College of Cardiology and the American Heart Association (ACC/AHA) released new guidelines on the diagnosis and treatment of elevated blood cholesterol and an equation for the assessment of cardiovascular risk in the American population, defined as the risk of a first arteriosclerotic cardiovascular disease (ASCVD) event in 10 years.[1, 2] While the ACC/AHA risk score considers non-fatal myocardial infarction, coronary heart disease death, as well as fatal or non-fatal stroke, the in Europe established European Society of Cardiology risk score [3] (ESC SCORE) only estimates risk of cardiovascular mortality.

External validations of the ACC/AHA risk score have been conducted in several U.S. cohorts [4-6] and in two European populations [7, 8] and substantial overestimation has been reported.[4-7] However, its performance has neither been tested in a German population nor has the ACC/AHA risk score been recalibrated to meet the underlying ASCVD risk in Germany.

We therefore sought to evaluate the ACC/AHA risk score prospectively in two population-based German cohorts with different background risk for ASCVD, the Monitoring of Trends and Determinants in Cardiovascular Disease (MONICA)/Cooperative Health Research in the Region of Augsburg (KORA) and the Heinz Nixdorf Recall (HNR) Studies. We recalibrated the ACC/AHA risk score using these German cohorts and compared performance of the original risk score with its recalibrated version and with the ESC SCORE.

Methods

Study populations

Cooperative Health Research in the Region of Augsburg Study

The KORA S3 and S4 surveys are population-based studies with baseline examinations in 1994–1995 and 1999–2001, respectively, including subjects from the city of Augsburg and the two adjacent counties located in the state of Bavaria in Southern Germany. Lifestyle factors and health conditions were assessed in computer-assisted personal interviews. Physical examinations were conducted by trained medical staff. Non-fatal stroke and non-fatal myocardial infarction were assessed by follow-up questionnaires in 2009 and validated by chart reviews or by contacting the subject`s primary care physician. Events of fatal stroke and fatal myocardial infarction were identified in 2011 by death certificates. In addition, information on non-fatal and fatal myocardial infarction was updated with information from the MONICA/KORA Myocardial Infarction Registry [9]. Subjects who had moved outside the study area were traced by address research; follow-up questionnaires were sent to their new address and self-reported non-fatal stroke or non-fatal myocardial infarction were validated by consulting the responsible local hospital. Detailed information on the KORA studies has been published elsewhere.[10]

For calculating the ACC/AHA risk score and in accordance with its empirical derivation,[2] subjects were excluded if they were outside the predefined age range of 40–79 years or had a previous history of congestive heart failure, myocardial infarction, or stroke prior to baseline. In addition, we excluded subjects with missing information on prevalent or incident events of myocardial infarction and stroke, as well as subjects with missing information on risk markers that were necessary to calculate the ACC/AHA risk score. Unlike Goff et al.[2], we were not able to exclude subjects with previous unrecognized myocardial infarction, percutaneous coronary intervention (PCI), coronary bypass surgery or atrial fibrillation, due to incompleteness or unavailability of this information in KORA S3 and S4. With these criteria, out of 9,116 KORA S3 and S4 subjects, 3,878 subjects had to be excluded, resulting in a final study population of 5,238 subjects (for details see Fig 1).

Fig 1

Flow chart of study populations.

Flow chart of the KORA and Heinz Nixdorf Recall study populations showing exclusion criteria in accordance with the ACC/AHA risk score.

In KORA, 486 subjects were lost-to-follow-up without information on incident ASCVD. To rule out possible bias in analysing the ACC/AHA risk score, we performed a drop-out analysis. Detailed description and results are shown in S1 Appendix.

Heinz Nixdorf Recall Study

The Heinz Nixdorf Recall study is a population-based prospective cohort study, designed to assess the predictive value of novel risk markers in addition to traditional cardiovascular risk factors. Participants were randomly selected from mandatory lists of residence from the three adjacent cities of Bochum, Essen and Mülheim, located in the Rhine-Ruhr-Area in Western Germany and enrolled between 2000 and 2003. Details of recruitment, study design, and risk factor assessment have been previously published.[11-13] Annually, questionnaires on the current state of health were sent out to all participants. In parallel, death certificates were regularly screened. Incident coronary events (myocardial infarction, sudden cardiac death and fatal coronary heart disease) and strokes were validated by review of all available hospital records and records of the attending physicians and adjudicated by an external endpoint committee. Fatal or non-fatal myocardial infarction was defined based on symptoms, electrocardiographic signs, cardiac enzymes, and necropsy. Stroke was defined as a focal neurological deficit over a period of >24hours of presumed cerebrovascular origin. In addition, cardiovascular mortality of any cause was classified by the statistical state office based on death certificate information according to the International Statistical Classification of Disease (ICD).

Exclusion criteria were applied according to those in KORA. From an overall HNR population of 4,814 subjects, 606 had to be excluded (thereof 101 without information on incident myocardial infarction or stroke), resulting in 4,208 eligible subjects (for details see Fig 1).

Ethical standards

All subjects signed written informed consent. For KORA, study protocols have been approved by the local Ethics Committee of the Bavarian Medical Association. The research protocol of the Heinz Nixdorf Recall study has been approved by the Ethics Committee of the Medical Faculty, University Duisburg-Essen.

The ACC/AHA risk score

The ACC/AHA risk score consists of sex- and race-specific equations estimating the individual 10-year risk of hard ASCVD events, defined as the first occurrence of a non-fatal myocardial infarction, coronary heart disease death, fatal or non-fatal stroke.[2] Risk factors included in the risk equations are age (years), total cholesterol (mg/dL), high-density lipoprotein (mg/dL), systolic blood pressure (mm Hg), blood pressure treatment (yes/no), current smoking (yes/no) and diabetes (yes/no). The ACC/AHA guidelines include different equations for Caucasians and non-Caucasians, hence, we used the equations for Caucasians for risk estimation in the two German cohorts. Details on the calculation of the ACC/AHA risk equations were derived from Goff et al. [2] An online calculator is available on http://www.cvriskcalculator.com/.

To consider differing background risk of the U.S. and Germany, we recalibrated the original ACC/AHA risk equations by Goff et al. [2] Using the pooled data set of the two German cohorts, we applied the recalibration technique described in Janssen et al., which corrects the risk score to comply with ‘calibration in the large’ [14]. This method calibrates the estimated risk values by adding the correction factor to the linear predictor within the risk equation. The original and recalibrated risk equations are summarized in S1 Table.

The ESC SCORE

Predicting 10-year risk of cardiovascular mortality, the ESC SCORE equation includes information on sex, age (years), total cholesterol (mmol/L), systolic blood pressure (mmHg) and current smoking (yes/no). Baseline survival was estimated separately for high-risk and low-risk countries. Since Germany is classified as a low-risk country, we used the corresponding equation. Calculations were performed as detailed in Conroy et al.[3]

For the ESC SCORE, events in the two study populations were re-categorized into fatal cardiovascular events with pre-determined ICD codes as defined in Conroy et al.[3] Exclusion criteria of the ESC SCORE differed from those of the ACC/AHA risk score and are illustrated in detail in S1 Fig.

Statistics

If not indicated otherwise, continuous variables are presented as median (first quartile, third quartile). Categorical variables are given as absolute number (percentage). Incidence rates were calculated as the number of incident events divided by the sum of person-years of all subjects at risk. Follow-up times of more than 10 years were censored.

To evaluate the risk scores, we observed discrimination and calibration performance of the risk scores for the pooled cohorts as well as separately by cohort. Discrimination performance of the risk scores was evaluated using Harrell’s concordance statistic (C-statistic) for survival data.[15, 16] Agreement of estimated risk and observed event frequency was assessed by a calibration plot categorizing subjects in deciles of estimated risk and comparing mean estimated risk versus average incidence per decile.[17, 18] In addition, the degree of miscalibration was assessed using following formula: Overestimation (%) = [(Estimated risk rate / Observed event rate) - 1] * 100. Positive numbers indicate an overestimation of the observed event rate, negative numbers stand for an underestimation.

Results regarding calibration and discrimination performance of the recalibrated ACC/AHA risk score were validated using 10-fold cross-validation. This method of internal validation ensures an unbiased estimation of the performance measures.

Statistical analysis was done in R version 3.0.2 using packages pROC, survival and survcomp.[19-22]

Results

Characteristics of study cohorts

The KORA study population consisted of 5,238 subjects aged 56.0 ± 9.7 (mean ± standard deviation) years, and 49% men (Table 1). During an average follow-up time of 8.6 ± 2.0 years, 383 (7.3%) incident ASCVD events occurred. From the HNR study, 4,208 subjects aged 59.0 ± 7.7 years (48% men) could be included. Of these, 271 (6.4%) subjects had a first incident ASCVD event during an average follow-up time of 9.0 ± 2.1 years. Comparing the risk profile of the two cohorts, we observed small differences regarding intake of antihypertensive drugs, intake of statins or fibrates and prevalence of diabetes. Incidence rates of ASCVD were 8.5 and 7.1 per 1,000 person years in KORA and in HNR, respectively. In both cohorts, incidence of ASCVD was more than two times higher in men compared to women.

Table 1

Basic description of study populations after applying exclusion criteria according to the ACC/AHA risk score.

Characteristica	KORA		HNR
	Men	Women	Men	Women
N (%)	2,584 (49)	2,654 (51)	2,005 (48)	2,203 (52)
Ageb (years)	56.4 ± 9.6	55.5 ± 9.7	58.8 ± 7.6	59.1 ± 7.7
Total cholesterol (mg/dL)	235 (208, 263)	233 (209, 264)	226 (202, 249)	232 (208, 260)
HDL cholesterol (mg/dL)	48 (40, 58)	60 (49, 72)	49 (42, 59)	64 (53, 75)
Systolic blood pressure (mmHg)	136 (125, 149)	128 (116, 144)	137 (125, 149)	125 (113,141)
Intake of antihypertensive drugs	486 (19)	583 (22)	604 (30)	696 (32)
Intake of statins or fibratesc	111 (4)	138 (5)	180 (10)	178 (9)
Current smoker	639 (25)	457 (17)	518 (26)	474 (22)
Diabetes	136 (5)	115 (4)	167 (8)	127 (6)
Incident ASCVD events	257 (10)	126 (5)	186 (9)	85 (4)
Fatal cardiovascular eventsd	119 (4)	56 (2)	49 (3)	25 (1)

N, Sample size

HDL, High-density lipoprotein

ASCVD, atherosclerotic cardiovascular disease

aDepicted are absolute numbers (percentage) for categorical and median (first quartile, third quartile) for continuous variables.

bAge (years) is shown as mean ± standard deviation.

cIn HNR, information on intake of fibrates or statins was only available in n = 1,861/2,071 men/women.

dFatal cardiovascular events according to Conroy et al.[3] after applying regarding exclusion criteria (Sample size for KORA men/women n = 2,805/2,950 and for HNR men/women n = 1,929/2,158).

Calibration

Table 2 provides, next to the observed and estimated event rates, a comparison of the magnitude of overestimation by the original and the recalibrated ACC/AHA risk score, as well as by the ESC SCORE.

Table 2

Overall overestimation by risk scores.

	KORA			HNR
	Total	Men	Women	Total	Men	Women
ACC/AHA risk score
- Original
Observed frequency [%]	7.3	9.9	4.7	6.4	9.3	3.9
Estimated risk [%]	10.0	14.2	5.9	10.7	15.1	6.6
Overestimation [%]	36.9	42.6	25.1	65.9	63.2	71.7
- Recalibrated
Observed frequency [%]	7.3	9.9	4.7	6.4	9.3	3.9
Estimated risk [%]	6.7	9.6	4.0	7.2	10.3	4.4
Overestimation [%]	-7.7	-3.3	-16.7	11.7	10.6	14.2
ESC SCORE
Observed frequency [%]	3.0	4.2	1.9	1.8	2.5	1.2
Estimated risk [%]	3.2	4.5	2.0	3.3	4.7	2.1
Overestimation [%]	5.3	5.3	5.3	84.6	86.6	80.8

The original ACC/AHA risk score overestimated risk by 43% in men and 25% in women in KORA and by 63% in men and 72% in women in HNR, respectively. Recalibration of the ACC/AHA risk score based on the observed event rate in both cohorts led to a substantially reduced miscalibration resulting in an underestimation of risk by 3% in men and 17% in women in KORA and an overestimation of risk by 11% in men and 14% in women in the HNR cohort.

The ESC SCORE overestimated observed event rates in KORA by 5% in men as well as by 5% in women, while in HNR overestimation amounted to 87% in men and 81% in women. In KORA, risk estimates and observed event frequencies therefore agreed better using the ESC SCORE than using the original or recalibrated ACC/AHA risk score, whereas in the HNR population, the recalibrated ACC/AHA risk score was the best calibrated score.

These findings were confirmed by the calibration plots (Fig 2) which give more insight into the complete risk range. For the pooled sample as well as separately by cohort, the calibration curves of the ACC/AHA risk score were deflected below the bisecting line, particularly in the highest risk deciles. The recalibrated ACC/AHA risk score showed considerably improved calibration over the complete risk range: for KORA, we detected slight underestimation and for HNR slight overestimation in the highest risk decile. The ESC SCORE revealed almost perfect calibration in KORA and overestimation in the upper risk categories in HNR.

Fig 2

Calibration plots.

Calibration plots of ACC/AHA risk score before (left side) and after (middle) recalibration and of ESC SCORE (right side). Part A shows calibration in the pooled sample of KORA and HNR. Part B shows calibration by cohort (KORA: filled circles, HNR: triangles). The solid line indicates perfect calibration of the risk score. Depicted are mean estimated risk versus mean observed frequency per decile of estimated risk with axes ranging from 0 to 0.5 (50%).

As the KORA cohort consisted of two individual surveys (S3 and S4) from different recruitment periods, we also investigated calibration performance of the ACC/AHA risk score separately for both. Risk was overestimated by the original risk score in both surveys, however, considerably more in KORA S4. Recalibration reduced miscalibration for both surveys (S2 Fig).

Discrimination

Discrimination analysis of the ACC/AHA risk score showed a C-statistic [95% confidence interval] of C = 0.76 [0.73, 0.79] in the pooled sample, C = 0.78 [0.73, 0.82] in the KORA cohort and C = 0.74 [0.68, 0.79] in the HNR cohort (Fig 3). As expected, these findings were identical to the discriminative performance of the recalibrated score. Estimating 10-year risk of cardiovascular mortality, the ESC SCORE showed better discrimination ability (pooled sample: C = 0.81 [0.75, 0.85], KORA: C = 0.82 [0.76, 0.87], HNR: C = 0.76 [0.65, 0.85]) than both the original and the recalibrated ACC/AHA risk score. All risk scores resulted in a better discriminative performance in KORA compared to HNR.

Fig 3

ROC curves.

ROC curves of ACC/AHA risk score before (left side) and after (middle) recalibration and of ESC SCORE (right side). Part A illustrates performance in the pooled sample of KORA and HNR, while part B shows performance separately for KORA (solid) and HNR (dashed). C = C-statistic [95% confidence interval].

Discussion

We found considerable overestimation of 10-year ASCVD risk by the original ACC/AHA risk score in two German population-based cohorts. This observation is in line with results from Kavousi et al. who detected consistent overestimation across the whole range of risk in a Dutch population.[7] Vaucher et al., who investigated the economic impact of the new ACC/AHA guidelines compared to the ESC guidelines in a Swiss population, observed a larger number of high-risk individuals when applying the ACC/AHA guidelines.[8] They suggested that differences in risk factor profiles between the U.S. and Europe could be the reason for this observation. Other studies found a similar overestimation when assessing the original Framingham risk score in European populations.[23-25] Given these observations, it is likely that the overestimation of risk that we found in our study when applying the original ACC/AHA score to a German population may at least in part be attributable to a different risk profile compared to the U.S population.

Nevertheless, we also have to consider the recruitment time of the studies included in the development of the ACC/AHA risk equations. Goff et al. incorporated study cohorts such as the Framingham original and offspring, the Atherosclerosis Risk in Communities (ARIC), the Coronary Artery Risk Development in Young Adults (CARDIA), and the Cardiovascular Health Study (CHS) cohort, most of which recruited their participants in the mid-1980s.[2] Changing risk profiles over time may contribute to risk overestimation when applying the ACC/AHA risk score to more contemporary populations.[26] This hypothesis is supported by our analysis wherein we found that the original ACC/AHA risk score performed best in the earliest recruited cohort KORA S3 (1994–95), had a poorer performance in the more recent cohort KORA S4 (1999–2001) and performed worst in HNR, the most recently assessed study population (2000–03). While diagnostic changes over time are one plausible explanation for such differences,[27] alterations in lifestyle factors and emerging risk markers, which may not have been sufficiently considered in the ACC/AHA risk score, would also explain the performance discrepancy of the ACC/AHA risk score.

We observed differences in the performance of the ACC/AHA risk score and the ESC SCORE between the KORA and the HNR study population. Different recruitment time points of HNR, KORA S3 and KORA S4 may be partly responsible for these differences, reflecting, in part, the decreasing incidence of cardiovascular events over the last 10 years.[28] In addition, we must consider regional diversity regarding cardiovascular risk within Germany.[29-31] We found a higher prescription rate of antihypertensives and lipid lowering drugs in HNR, possibly explaining the slightly lower incidence rate of ASCVD in the HNR population.

The different performance of the ESC SCORE in HNR and KORA is of particular interest, as an independent earlier survey within the KORA study (MONICA/KORA S1, 1984/85) contributed to the development of the ESC SCORE as the only population sample from Germany.[3] This may also explain the superior performance of the ESC SCORE in KORA compared to HNR. Nevertheless, such discrepancies in an established European risk score demonstrate the necessity of reconsidering the ESC SCORE regarding included risk factors and the requirement of including contemporary studies from Germany, such as HNR.

In our approach we focused on the recalibration of the ACC/AHA risk score, which means the evaluation and reassessment of the original risk score by using the originally included risk markers. However, recalibration does not consider the possibility of evaluating additional risk factors in the original score, such as family history of cardiovascular events. We would like to encourage future studies to include such risk factors to potentially increase the performance of the recalibrated ACC/AHA risk score.

Finally, we cannot rule out different ascertainment strategies for fatal and non-fatal ASCVD events in HNR and KORA as an additional possible source for the observed differences in the performance of the risk scores.

Limitations

One minor limitation of this study is the incomplete follow-up (follow-up time of less than 8 years) on non-fatal stroke in n = 278 subjects of the KORA survey S4. Although we do have information on non-existence of myocardial infarction and fatal stroke events in these subjects, we may have missed non-fatal stroke events. However, considering the small number of non-fatal stroke events after 8 years of follow-up in KORA survey S3 (event rate 0.5%), we assume that this incomplete follow-up in KORA survey S4 did not affect our results considerably.

We were not able to exclude subjects with previous unrecognized myocardial infarction, percutaneous coronary intervention (PCI), coronary bypass surgery or atrial fibrillation as it was done in the original version of the ACC/AHA risk score. Although unlikely, it is possible that these exclusions might have improved performance of the uncalibrated risk equations.

Our study compares the ACC/AHA risk score with the ESC SCORE, however, we did not consider a comparison to the only other Germany-specific risk score, the Prospective Cardiovascular Münster (PROCAM) score. This score is based on data from employees of companies and local government authorities and offers risk equations separate for major coronary events and stroke.[32] Although the PROCAM score showed good performance within the PROCAM cohort, one must question whether the development of a risk score only based on employed individuals can be representative for the general German population. For this reason, we have not considered its incorporation in our analysis.

Conclusions

The recalibrated ACC/AHA risk score showed improved risk prediction as compared to the original ACC/AHA risk score in both German cohorts. Predicting only cardiovascular mortality, discrimination performance of the commonly used ESC SCORE remained somewhat superior to the ACC/AHA risk score. Yet non-fatal events are of increasing importance given the declining case-fatality of acute myocardial infarction. Considering this advantage, the recalibrated ACC/AHA risk score provides a meaningful tool for estimating 10-year risk of fatal and non-fatal cardiovascular disease in the German population.

Drop-out analysis.

(PDF)

Click here for additional data file.

Flow chart of study populations showing exclusion criteria in accordance with the ESC SCORE.

(PDF)

Click here for additional data file.

Calibration plot by KORA survey.

Calibration plot of the ACC/AHA risk score before (left) and after (right) recalibration separately for KORA S3 (filled circles) and KORA S4 (triangles).

(PDF)

Click here for additional data file.

Calculation of ACC/AHA risk score before and after recalibration.

(PDF)

Click here for additional data file.