Prediction of coronary heart disease risk in a general, pre-diabetic, and diabetic population during 10 years of follow-up: accuracy of the Framingham, SCORE, and UKPDS risk functions: The Hoorn Study.

OBJECTIVE: To test the validity of the Framingham, Systematic Coronary Risk Evaluation (SCORE), and UK Prospective Diabetes Study (UKPDS) risk function in the prediction of risk of coronary heart disease (CHD) in populations with normal glucose tolerance (NGT), intermediate hyperglycemia, and type 2 diabetes. RESEARCH DESIGN AND METHODS: Calibration and discrimination of the three prediction models were tested using prospective data for 1,482 Caucasian men and women, 50-75 years of age, who participated in the Hoorn Study. All analyses were stratified by glucose status.
RESULTS: During 10 years of follow-up, a total of 197 CHD events, of which 43 were fatal, were observed in this population, with the highest percentage of first CHD events in the diabetic group. The Framingham and UKPDS prediction models overestimated the risk of first CHD event in all glucose tolerance groups. Overall, the prediction models had a low to moderate discriminatory capacity. The SCORE risk function was the best predictor of fatal CHD events in the group with NGT (area under the receiver operating characteristic curve 0.79 [95% CI 0.70-0.87]), whereas the UKPDS performed better in the intermediate hyperglycemia group (0.84 [0.74-0.94]) in the estimation of fatal CHD risk. After exclusion of known diabetic patients, all prediction models had a higher discriminatory ability in the group with diabetes.
CONCLUSIONS: The use of the Framingham function for prediction of the first CHD event is likely to overestimate an individual's absolute CHD risk. In CHD prevention, application of the SCORE and UKPDS functions might be useful in the absence of a more valid tool.

Coronary heart disease (CHD) is a leading cause of mortality in the world, and projections show that this will still be the case in 2025 (1). Awareness of the public health relevance of the disease has brought about an increasing body of evidence on interventions that can effectively reduce the incidence of CHD (2,3). A key component for effective targeting of these interventions is the assessment of an individual's absolute risk of developing CHD within a defined time period.

Risk of CHD can be assessed using several methods that combine values for different risk factors to produce a quantitative risk estimate. These risk calculation algorithms are derived from long-term prospective cohort studies. The most widely used models are based on data from the Framingham Heart Study for first CHD event (4,5). More recently, the Systematic Coronary Risk Evaluation (SCORE) Project developed a risk function for estimation of fatal CHD derived from European data (6). The UK Prospective Diabetes Study (UKPDS) Group developed a diabetes-specific algorithm for first CHD event, acknowledging the fact that patients with type 2 diabetes have an increased risk for CHD (7). Because of this increased risk for CHD, models derived from diabetic populations are expected to offer better predictive accuracy of CHD morbidity and mortality in individuals with diabetes than general population–based models.

An a priori assessment of the accuracy and validity of a risk algorithm should be performed when one is applying it to a population different from the one in which it was developed. Thus, the performance of the Framingham, SCORE, and UKPDS risk algorithms in predicting CHD events in both U.S. and European populations has been reported (8–14). The accuracy of these algorithms has not been prospectively evaluated in a population at high risk for developing diabetes, such as individuals with intermediate hyperglycemia, who have been identified as having an increased CHD risk compared with that of normoglycemic individuals or have even been thought to have the same risk as individuals with established diabetes (15). The purpose of the present study was to validate and compare results from the Framingham, SCORE, and UKPDS risk functions in predicting CHD risk of individuals with normal glucose tolerance (NGT), intermediate hyperglycemia (i.e., impaired glucose tolerance and/or impaired fasting glucose), and diabetes, using data from the population-based Hoorn Study cohort.

RESEARCH DESIGN AND METHODS

The Hoorn Study is a population-based cohort study of glucose metabolism of 2,484 Dutch Caucasian men and women, aged 50–75 years at study initiation in 1989. The study design and population have been described in detail elsewhere (16).

For the current study, we included participants with NGT (n = 1,791), participants with intermediate hyperglycemia (n = 423), and participants with type 2 diabetes (n = 255) at baseline. After exclusion of individuals with a previous history of cardiovascular disease at baseline (NGT n = 294, intermediate hyperglycemia n = 99, and diabetes n = 77) and of individuals with missing values for any of the predictor variables (NGT n = 8, intermediate hyperglycemia n = 3, and diabetes n = 10) or outcome variables (NGT n = 364, intermediate hyperglycemia n = 89, and diabetes n = 43), 1,125 individuals with NGT, 232 individuals with intermediate hyperglycemia, and 125 individuals with diabetes remained for the current analyses. All participants provided written informed consent. Ethics approval for the study was obtained from the Ethics Review Committee of the VU University Medical Center Amsterdam.

Measurements

According to baseline levels of fasting plasma glucose and 2 h after an oral glucose tolerance test, participants were classified into NGT, intermediate hyperglycemia, or diabetes groups, which were defined according to the 2006 criteria of the World Health Organization (17). Individuals who were already being treated for diabetes with insulin, oral hypoglycemic agents, or a physician-prescribed diet were categorized as known diabetic subjects, irrespective of their glucose levels.

Weight and height were measured in barefoot participants wearing light clothing. BMI was calculated as weight in kilograms divided by the square of height in meters. Blood pressure was measured in a sitting position after 5 min of rest using a random-zero sphygmomanometer (Hawksley-Gelman, Lancing, Sussex, U.K.). Serum total and HDL cholesterol levels were measured using an enzymatic technique (Boehringer-Mannheim, Mannheim, Germany) as described elsewhere (18). A1C was determined by ion-exchange high-performance liquid chromatography with a Modular Diabetes Monitoring System (Bio-Rad, Veenendaal, the Netherlands). Information about cigarette smoking, diabetes duration in those with a diagnosis of diabetes, ethnicity, medication use, history of cardiovascular diseases, and family history of myocardial infarction was assessed by self-administered questionnaires.

Primary end points defined for this study were CHD events that had occurred until 1 January 2000. Information on fatal and nonfatal CHD events was classified using medical records of general practitioners and the local hospital. Morbidity and fatal events were coded according to the ICD-9. CHD was defined as fatal and nonfatal ischemic heart disease and sudden death (ICD-9 codes 410–414, 427.4, 427.5, and 798).

Statistical analysis

Baseline characteristics are presented as means ± SD, median (interquartile range) in case of a skewed distribution, or percentages according to glucose status group. Estimated 10-year first CHD risk was calculated for each participant using the Framingham (5) and UKPDS (7) algorithms. Estimated 10-year risk of fatal CHD was calculated by means of the SCORE risk function developed for the low-risk region of Europe. To estimate the risk in the diabetic subgroup, the predicted risk by the SCORE algorithm was multiplied by 2 for men and by 4 for women, as recommended by Conroy et al. (6).

The predictive accuracy of the three risk functions was estimated using calibration (the ability to predict the number of observed events during follow-up) and discrimination appraisals (the ability to distinguish between those who experience a CHD event during follow-up from those who do not). These two approaches have been described in detail previously (19).

Calibration of the model was visually checked by plotting the predicted probabilities estimated by the prediction models against the observed proportion of first CHD events (Framingham and UKPDS) and fatal CHD events (SCORE). Participants were grouped into quintiles of predicted CHD risk within 10 years of follow-up. Plots were created for the three glucose status subgroups. The discriminatory ability of the models was assessed by calculating the area under the receiver operating characteristic curve (AUROC) using the 10-year risk estimates predicted by the three models. The discriminatory power of models is graded as low for an AUROC between 0.5 and 0.7, moderate between 0.7 and 0.9, and high if >0.9 (20).

Because the UKPDS risk function is derived from a cohort of patients with newly diagnosed diabetes and the Framingham and SCORE functions do not include information about diabetes duration, sensitivity analyses were carried out to test the discriminatory ability of the risk algorithms in predicting first or fatal CHD event in individuals with screening-detected diabetes only. Furthermore, because family history of myocardial infarction is a strong predictor of the incidence of CHD and is part of usual risk assessment in general practice (21), we tested whether the addition of family history of myocardial infarction to the prediction formulas improved risk estimation of the first CHD event. Improvements in AUROCs and reclassification improvements were examined and tested for significance using methods described previously (22,23). All statistical analyses were performed using SPSS 15.0 for Windows (SPSS, Chicago, IL).

RESULTS

Table 1 presents baseline and end point characteristics stratified by glucose status subgroup. Antihypertensive and lipid-lowering medication use was highest among diabetic subjects. A total of 197 first CHD events, of which 43 were fatal, were observed in this population. A higher percentage of first CHD events occurred in participants with diabetes compared with participants categorized in the NGT or intermediate hyperglycemia subgroup.

Table 1

Baseline characteristics of the study population stratified by glucose status group

	NGT group	Intermediate hyperglycemia group	Screening-detected diabetic group	Known diabetic group
n	1,125	232	85	40
Age (years)	60.3 ± 7.1	63.2 ± 7.3	64.5 ± 6.9	64.9 ± 7.0
Men (%)	45.2	47.2	54.0	46.3
BMI (kg/m2)	25.9 ± 3.1	27.5 ± 3.6	28.6 ± 4.1	27.6 ± 4.2
Systolic blood pressure (mmHg)	131.6 ± 19.2	143.3 ± 20.4	148.3 ± 21.3	143.2 ± 21.2
Diastolic blood pressure (mmHg)	81.4 ± 10.3	84 ± 9.8	85.8 ± 11.7	83.2 ± 10.4
Total cholesterol (mmol/l)	6.6 ± 1.2	6.7 ± 1.0	6.7 ± 1.4	6.5 ± 1.4
HDL cholesterol (mmol/l)	1.4 ± 0.4	1.3 ± 0.3	1.2 ± 0.3	1.2 ± 0.3
LDL cholesterol (mmol/l)	4.6 ± 1.1	4.6 ± 1.0	4.5 ± 1.2	4.2 ± 0.9
A1C (%)	5.3 ± 0.5	6.7 ± 1.0	6.4 ± 1.7	7.6 ± 1.6
Diabetes duration (years)	—	—	—	6.1 (2.0–11.2)
Current smokers (%)	32.6	24.5	25.3	17.1
Antihypertensive medication (%)	10.0	19.3	25.3	29.3
Lipid-lowering medication (%)	0.7	0.4	1.1	2.4
Family history of myocardial infarction	185 (16.5)	31 (13.4)	15 (17.6)	4 (10)
First CHD	108 (9.6)	24 (10.3)	16 (18.8)	12 (30.0)
Fatal CHD	27 (2.4)	6 (2.6)	8 (9.4)	2 (5)
Incidence rate*
First CHD	1,170.6	1,310.9	2,532.8	4,470.9
Fatal CHD	292.6	327.7	1,266.4	745.2

Values are presented as means ± SD, median (interquartile range), absolute numbers, or n (%).

*The incidence rate is expressed as the number of new cases per 100,000 person-years.

Figure 1 depicts the calibration plots for the prediction of first and fatal CHD events. The Framingham and UKPDS prediction models showed an overestimation of the risk of the first CHD event, mainly in the upper quintiles of predicted risk in the NGT, the intermediate hyperglycemia, and the diabetes subgroup. The estimation of fatal CHD events by the SCORE algorithm was fair in both the NGT and intermediate hyperglycemia group but less precise in the diabetic group.

Figure 1

Observed and predicted incidence of first CHD events (A–C) and fatal CHD events (D–F) in quintiles of predicted risk estimated by the Framingham (□), SCORE (○), and UKPDS (▲) risk function according to glucose status group. IH, intermediate hyperglycemia; DM, type 2 diabetes.

The discriminatory results from the three models, applied to the Hoorn subpopulations, are presented in Table 2, showing the AUROCs and 95% CIs. The Framingham prediction formula showed the lowest AUROCs among all glucose status groups in estimation of first CHD risk. The SCORE risk function showed the highest area under the curve when fatal CHD risk in the NGT subgroup (AUROC 0.79 [95% CI 0.70–0.87]) and in the diabetic subgroup (0.74 [0.56–0.93]) was estimated. For the intermediate hyperglycemia group, moderate discriminatory ability (AUROC >0.70) was only seen for Framingham (0.76 [0.65–0.88]) and UKPDS (0.84 [0.74 – 0.94]) for fatal CHD risk. In the diabetic subgroup, all prediction models had low discriminatory ability for estimating the risk of first CHD event, whereas SCORE and UKPDS had moderate ability to identify individuals with a high risk for a fatal CHD event. The UKPDS risk model, derived from a diabetic population, showed a slightly higher area under the curve in a population with NGT (0.71 [0.66–0.75]) and in the population with intermediate hyperglycemia (0.70 [0.60–0.80]) compared with the diabetic population (0.66 [0.56–0.78]). Overall, all prediction models showed better discrimination when a fatal CHD event was used as the predicted outcome.

Table 2

Discriminatory ability of the Framingham, SCORE, and UKPDS risk functions

	NGT group	Intermediate hyperglycemia group	Diabetic group
First CHD
Framingham	0.68 (0.63–0.74)	0.60 (0.47–0.74)	0.63 (0.50–0.76)
SCORE	0.71 (0.66–0.76)	0.70 (0.60–0.79)	0.66 (0.54–0.79)
UKPDS	0.71 (0.66–0.75)	0.70 (0.60–0.80)	0.66 (0.56–0.78)
Fatal CHD
Framingham	0.71 (0.61–0.82)	0.76 (0.65–0.88)	0.61 (0.37–0.86)
SCORE	0.79 (0.70–0.87)	0.70 (0.50–0.89)	0.74 (0.56–0.93)
UKPDS	0.77 (0.68–0.86)	0.84 (0.74–0.94)	0.72 (0.55–0.89)

Data are AUROC (95% CI).

When the discriminatory ability of the three risk functions to estimate risk of a first CHD event in individuals with screening-detected diabetes was analyzed, all risk functions showed higher areas under the curve in this group compared with those in the group in which individuals with screening-detected diabetes and known diabetes were combined (Framingham 0.74 [95% CI 0.59–0.89], SCORE 0.79 [0.66–0.92], and UKPDS 0.75 [0.63–0.87]). Similar results were seen for estimations of a fatal CHD event (Framingham 0.73 [0.50–0.95], SCORE 0.82 [0.67–0.97], and UKPDS 0.83 [0.68–0.97]). The addition of family history of myocardial infarction slightly improved most risk algorithms in prediction of the risk of a first CHD event although the changes were not statistically significant.

CONCLUSIONS

In this study, we compared the performance of two risk functions designed for the general population and one diabetes-specific risk function in predicting nonfatal and/or fatal CHD risk in a general, pre-diabetic, and diabetic population during 10 years of follow-up. With respect to the agreement between the predicted and observed estimates, the Framingham and UKPDS risk function overall overestimated the actual observed CHD incidence rate in the three subgroups. Regarding the risk of first CHD, the Framingham algorithm had low ability to discriminate in all subgroups except for the group consisting of patients with screening-detected diabetes in whom the discriminatory ability was moderate. The UKPDS function had moderate ability to identify those with high risk for a first CHD event in the NGT and intermediate hyperglycemia groups and in patients with screening-detected diabetes but low ability in the group in which patients with screening-detected and known diabetes were combined. The SCORE algorithm for the prediction of fatal CHD has a moderate ability in all subgroups. Although the Framingham and UKPDS risk functions were designed to estimate first CHD in the general population and the diabetic population, respectively, both functions performed better in estimating fatal CHD than the SCORE risk function for fatal CHD with the highest discriminatory ability being observed for the UKPDS risk function in the intermediate hyperglycemia group.

The results of our study do not support the findings in previous studies in which the CHD risk in a population with diabetes was underestimated by the Framingham (10,14) and UKPDS risk functions (12). Previous research also showed an overestimation of CHD risk predicted by the UKPDS risk function during 5 years of follow-up (24), the SCORE risk function during 10 years of follow-up (11), and the Framingham risk function in European populations (8,9). Recently, the performance of the Framingham and UKPDS risk functions for estimation of cardiovascular disease risk has been tested, and both functions were found to overestimate the cardiovascular disease risk and to be moderately effective in a normoglycemic, pre-diabetic, and diabetic population (25). These results are comparable to the predictive accuracy of the CHD risk functions in our study despite the different criteria used to define the three glucose status groups.

Some possible reasons for the overestimation of CHD risk by the Framingham and UKPDS prediction models in our study population can be suggested. Prediction of the risk of CHD, calculated by the Framingham function, might result in less precise CHD estimates in a population well treated for cardiovascular risk factors, as is the case with the Hoorn population. This phenomenon may also contribute to the better prediction in patients with newly diagnosed diabetes compared with those with known diabetes. In addition, it is suggested that risk estimation might be more accurate in individuals with fewer risk factors (4). Indeed, in the current study, a larger overestimation of the actual observed CHD events was seen in individuals in the higher quintiles of estimated CHD risk. Regarding the SCORE risk function for estimating fatal CHD events, the estimation could have been less precise because of the small number of observed fatal CHD events in the current study. In addition, participants with missing information on CHD events were excluded, and a selection bias could have occurred. The population used for the current study consists of Caucasian men and women. Because performance of the risk functions might differ between different ethnic groups, the results of our study cannot be immediately generalized to populations of other ethnic origin. Furthermore, the UKPDS risk function was based on patients with newly diagnosed diabetes, and our results suggested better performance of the risk algorithms when they were applied solely to patients with newly detected diabetes. The SCORE and Framingham risk functions also showed better discriminatory ability in patients with screening-detected diabetes. Values of risk factors used for the development of the UKPDS prediction model were based on measurements taken at 1 or 2 years after diagnosis of diabetes. At that time treatment has changed, which resulted in altered risk profiles for most patients. In our diabetic subgroup, 68% of the participants had the diagnosis of diabetes at baseline and risk factor levels at the baseline measurement were used for all analyses. The different time of measurement could be a reason for the overestimation of the risk of first CHD calculated by the UKPDS risk engine.

In summary, we found that the use of the Framingham function in the prediction of first CHD event in all participants in the Hoorn cohort was likely to overestimate an individual's absolute CHD risk. From a clinical perspective, this overestimation may lead to improper targeting of preventive strategies because of the large number of patients with false-positive results who are identified for treatment. From a health policy point of view, this may cause incorrect allocation of health resources. The equation might be used to estimate potential relative reduction in CHD risks. To aid in CHD prevention, application of the SCORE risk function in diabetic patients and application of the SCORE and UKPDS risk functions in individuals with normal glucose metabolism or intermediate hyperglycemia might prove useful in the absence of a more valid tool.