Performance of the UK Prospective Diabetes Study Risk Engine and the Framingham Risk Equations in Estimating Cardiovascular Disease in the EPIC- Norfolk Cohort.

OBJECTIVE: The purpose of this study was to examine the performance of the UK Prospective Diabetes Study (UKPDS) Risk Engine (version 3) and the Framingham risk equations (2008) in estimating cardiovascular disease (CVD) incidence in three populations: 1) individuals with known diabetes; 2) individuals with nondiabetic hyperglycemia, defined as A1C >or=6.0%; and 3) individuals with normoglycemia defined as A1C <6.0%. RESEARCH DESIGN AND METHODS: This was a population-based prospective cohort (European Prospective Investigation of Cancer-Norfolk). Participants aged 40-79 years recruited from U.K. general practices attended a health examination (1993-1998) and were followed for CVD events/death until April 2007. CVD risk estimates were calculated for 10,137 individuals.
RESULTS: Over 10.1 years, there were 69 CVD events in the diabetes group (25.4%), 160 in the hyperglycemia group (17.7%), and 732 in the normoglycemia group (8.2%). Estimated CVD 10-year risk in the diabetes group was 33 and 37% using the UKPDS and Framingham equations, respectively. In the hyperglycemia group, estimated CVD risks were 31 and 22%, respectively, and for the normoglycemia group risks were 20 and 14%, respectively. There were no significant differences in the ability of the risk equations to discriminate between individuals at different risk of CVD events in each subgroup; both equations overestimated CVD risk. The Framingham equations performed better in the hyperglycemia and normoglycemia groups as they did not overestimate risk as much as the UKPDS Risk Engine, and they classified more participants correctly.
CONCLUSIONS: Both the UKPDS Risk Engine and Framingham risk equations were moderately effective at ranking individuals and are therefore suitable for resource prioritization. However, both overestimated true risk, which is important when one is using scores to communicate prognostic information to individuals.

Individuals with type 2 diabetes have a two to four times increased risk of cardiovascular disease (CVD) compared with those without diabetes (1). Multifactorial interventions aimed to reduce hyperglycemia, hypertension, and hypercholesterolemia are effective for reducing the risk of cardiovascular and microvascular events in diabetic individuals (2,3). Multivariate equations such as the Framingham equations are used to estimate CVD risk to target therapy to those with the highest absolute risk and to provide patients and practitioners with prognostic information. However, although some studies have concluded that the Framingham risk equations for estimating CVD risk provide acceptable results when applied to populations outside North America (4), others have suggested that they are not applicable in those with a particularly low or high risk (5), including individuals with diabetes (6). The UK Prospective Diabetes Study (UKPDS) Risk Engine is a type 2 diabetes-specific risk calculator that includes A1C as well as traditional CVD risk factors. Version 2 of the Risk Engine estimates coronary heart disease risk and stroke risk separately. In version 3, equations have been derived that estimate CVD risk directly (7). This novel risk equation has been validated in the Collaborative Atorvastatin Diabetes Study (CARDS) cohort (8), which was a primary prevention trial, and showed good predictive ability. The CARDS trial cohort is not necessarily as widely generalizable as a true population-based sample. Thus, in this article we examined the performance of the UKPDS Risk Engine (version 3) and the Framingham risk equations (2008) in estimating CVD incidence in three population subgroups: 1) individuals with known diabetes; 2) individuals with nondiabetic hyperglycemia (A1C ≥6.0%); and 3) individuals with A1C <6.0% (normoglycemia).

RESEARCH DESIGN AND METHODS

European Prospective Investigation of Cancer (EPIC)-Norfolk is a prospective cohort study in which men and women aged 40–79 years were recruited from general practices in the Norfolk region of the U.K. Full details of the population are reported elsewhere (9). In brief, between 1993 and 1998, 25,639 individuals attended a baseline health examination. This included anthropometric and blood pressure measurements and completion of a general health questionnaire, with questions on personal and family history of disease, medication, and lifestyle factors. Participants were asked to indicate whether they were a current smoker, ex-smoker, or never smoker. They were also asked whether a doctor had ever told them that they had any of the conditions contained in a list that included diabetes, heart attack, and stroke. In addition, baseline diabetes status was also ascertained by 1) a self-report of diabetes medication; 2) diabetes medication brought to the baseline health check; 3) indication of a modification in diet in the past year because of diabetes; or 4) indication of following a diet for diabetes. Nonfasting blood samples were obtained, and starting in 1995 when funding became available, A1C was measured on fresh EDTA blood samples using high-performance liquid chromatography (Diamat automated glycated hemoglobin analyzer; Bio-Rad, Hemel Hempstead, U.K.).

The population in the Norfolk area is healthier than the general U.K. population with a standardized mortality ratio of 94 (source: Office for National Statistics). However, the EPIC-Norfolk cohort is similar to a nationally representative sample for anthropometric variables, blood pressure, and serum lipids (9).

We report results for follow-up to April 2007. Participants were followed for a median of 10.1 years. All EPIC-Norfolk participants were flagged for death certification at the Office of National Statistics, and vital status was obtained for the entire cohort. Trained nosologists coded death certificates according to the ICD-9 or ICD-10. Cardiovascular death (stroke, coronary heart disease, peripheral vascular disease, and other vascular causes) was defined in those whose underlying cause of death was coded as ICD-9 400–448 or ICD-10 I10–I79. Participants admitted to a hospital were identified by their National Health Service number. Hospitals were linked to the East Norfolk Health Authority database, which identifies all hospital contacts throughout England and Wales for Norfolk residents. Participants were identified as having a CVD event during follow-up if they had a hospital admission and/or died with CVD as the underlying cause. Previous validation studies in our cohort indicated high specificity of such case ascertainment (10).

Estimation of cardiovascular risk

The 10-year absolute risk of CVD was estimated for each participant using the UKPDS Risk Engine version 3.0 (7). This is a type 2 diabetes-specific risk assessment tool that defines CVD as the first event to occur of fatal or nonfatal myocardial infarction, sudden cardiac death, other incident ischemic heart disease, fatal or nonfatal stroke, and peripheral vascular disease death. Similarly, we applied the Framingham CVD risk equations (2008) (11) for each participant, which defines CVD as the first event to occur of myocardial infarction (including silent and unrecognized myocardial infarction), death from coronary heart disease (CHD) (sudden or nonsudden), CHD (myocardial infarction and CHD death plus angina pectoris and coronary insufficiency), stroke, transient ischemic attack, congestive heart failure, and peripheral vascular disease.

Statistical analysis

We excluded individuals with self-reported CVD at baseline (n = 1,106) and those with missing values for one or more of the variables (ethnicity, smoking status, total cholesterol, HDL cholesterol, systolic blood pressure, and A1C) used to calculate the Framingham and UKPDS Risk Engine CVD risk estimates (n = 548). Because A1C measurement started approximately half-way through the data collection period, only 42% of the original sample had A1C values at baseline.

Baseline characteristics were summarized separately in population subgroups using means and percentages. The subgroups encompassed 1) individuals with known diabetes; 2) individuals with nondiabetic hyperglycemia, defined as A1C ≥6.0%; and 3) individuals with A1C <6.0% (normoglycemia). We calculated the observed mean CVD risk and the estimated CVD risk using the UKPDS Risk Engine and Framingham equations. We examined their performance by 1) comparing the area under the receiver operating characteristic curve (aROC) using a nonparametric algorithm (12) to assess discrimination, 2) computing a Bayes information criterion (BIC) statistic to assess the global fit of the models, and 3) examining the proportion of men and women who would be reclassified into higher- or lower-risk categories between the two equations, using the Net Reclassification Improvement (NRI) statistic (13). We assessed the calibration of each equation using a goodness-of-fit test statistic. All analyses were performed in the whole EPIC-Norfolk population and separately by sex. Sensitivity analyses were performed to examine possible differences in baseline characteristics between participants with and without A1C data.

All analyses were completed using Stata (version 10.0; StataCorp, College Station, TX). The EPIC-Norfolk study was approved by the Norfolk Local Research Ethics Committee, and participants gave written consent before the first health check.

RESULTS

The dataset included 4,424 men and 5,713 women for whom we had complete data available, including A1C. The cumulative incidence rate of CVD was 9.8 per 1,000 person-years.

Baseline characteristics for the EPIC-Norfolk cohort, by population subgroup, are shown in Table 1. Individuals with prevalent diabetes had the highest mean age, and there was a higher proportion of men compared with other groups. Similarly, individuals with diabetes had the highest mean BMI, systolic blood pressure, and A1C and were the most likely to report statin use but had the lowest proportion of current smokers. Individuals with nondiabetic hyperglycemia had the highest mean total cholesterol and LDL cholesterol.

Table 1

Baseline characteristics by population subgroup and incident CVD events, EPIC-Norfolk cohort, U.K., 1993–2007

Characteristic	Individuals with prevalent diabetes	Individuals with nondiabetic hyperglycemia	Normoglycemic individuals	P value for difference†
n	272	906	8,959
Mean age (years)	62.8 ± 8.6	62.6 ± 8.4	56.6 ± 9.6	<0.001
Women	129 (47.4)	498 (55.0)	5,086 (56.8)	0.006
Social class*
I to III nonmanual	156 (58.9)	497 (56.0)	5,485 (61.2)
III manual to V	109 (41.1)	391 (44.0)	3,329 (37.2)	0.003
Caucasian	272 (100.0)	904 (99.8)	8,919 (99.6)	0.990
Mean BMI (kg/m2)	27.8 ± 5.0	27.5 ± 4.5	26.0 ± 3.8	<0.001
Mean total cholesterol (mmol/l)	6.0 ± 1.2	6.4 ± 1.2	6.1 ± 1.1	<0.001
Mean HDL (mmol/l)	1.4 ± 0.4	1.4 ± 0.4	1.5 ± 0.4	0.457
Mean LDL (mmol/l)	3.8 ± 1.0	4.1 ± 1.1	3.9 ± 1.0	<0.001
Mean systolic blood pressure (mmHg)	141.4 ± 19.0	140.8 ± 17.6	133.5 ± 17.9	<0.001
Statin use	10 (3.7)	18 (2.0)	94 (1.1)	<0.001
Current smoker	23 (8.5)	155 (17.1)	1,028 (11.5)	<0.001
Mean A1C (%)	7.5 ± 2.0	6.4 ± 0.9	5.1 ± 0.5	<0.001
CVD events	69 (25.4)	160 (17.7)	732 (8.2)	<0.001

Data are means ± SD or n (%). n = 10,137.

*Numbers may not add up to total due to missing values.

†Groups were compared using one-way ANOVA for continuous variables and χ2 tests for categorical variables.

Over a median of 10.1 years of follow-up, there were 69 CVD events in the 272 individuals with diabetes (25.4%), 160 in the 906 with nondiabetic hyperglycemia (17.7%), and 732 in the 8,959 with normoglycemia (8.2%) (Table 1). The estimated CVD 10-year risk in individuals with diabetes was 33 and 37% using the UKPDS and Framingham equations, respectively (Table 2). In the hyperglycemia group, estimated CVD risk was 31 and 22%, respectively, and in the normoglycemia group, estimated CVD risk was 20 and 14%, respectively.

Table 2

Actual and estimated CVD risk, aROC curve, and BIC for the UKPDS and Framingham CVD risk equations in each population subgroup, EPIC-Norfolk cohort, U.K. 1993–2007

	Individuals with prevalent diabetes	Individuals with nondiabetic hyperglycemia	Normoglycemic individuals
n	272	906	8,959
Actual CVD event rate	25.4	17.7	8.2
Estimated CVD 10-year risk: UKPDS Risk Engine [% (95% CI)]	33.2 (28.1–38.5)	30.5 (25.9–35.4)	20.4 (17.0–24.2)
Estimated CVD 10-year risk: Framingham equations (%)	36.7	22.3	14.4
aROC (95% CI) for the UKPDS Risk Engine	0.72 (0.65–0.78)	0.68 (0.63–0.72)	0.77 (0.76–0.79)
aROC (95% CI) for the Framingham equations	0.73 (0.66–0.78)	0.66 (0.62–0.71)	0.77 (0.76–0.79)
Sensitivity/specificity of the UKPDS Risk Engine*	0.94/0.31	0.94/0.22	0.97/0.15
Sensitivity/specificity of the Framingham equations*	0.86/0.30	0.90/0.26	0.96/0.20
BIC for the UKPDS Risk Engine	288	807	4,444
BIC for the Framingham equations	285	816	4,405

*Using a cutoff point of 0.20, e.g., 20% absolute risk of a CVD event in the next 10 years.

The aROC for individuals with diabetes in EPIC-Norfolk was 0.72 for the UKPDS Risk Engine and 0.73 for the Framingham equations (Table 2). There was no statistically significant difference in their discrimination (P = 0.58). Similarly, the aROC for individuals with nondiabetic hyperglycemia was 0.68 for the UKPDS Risk Engine and 0.66 for the Framingham equations, with no significant difference in discrimination (P = 0.16). For normoglycemic individuals, the aROC for the UKPDS Risk Engine was 0.77 and for the Framingham equations was 0.77, with no evidence of a difference in the ability of the equations to discriminate between those who had a CVD event and those who did not (P = 0.38). The shapes of each set of ROC curves were roughly similar in each subgroup (data not shown). The BIC value was similar for both risk equations in each population subgroup, indicating good global model fit.

Both the Framingham risk equations and the UKPDS Risk Engine had good to excellent ability to correctly identify individuals who would develop CVD using a cutoff point of 20% in all three subgroups (Table 2). In the diabetes group, for example, the sensitivity was 0.94 for the Risk Engine and 0.86 for the Framingham equations. However, both equations had poor specificity. The specificity was highest in the diabetes group (31 and 30% for the Risk Engine and Framingham equations, respectively) and lowest in the normoglycemia group, with the Risk Engine only achieving 15%.

Reclassifications are summarized in Table 3. The NRI refers to the net gain in correct reclassification and was calculated using notation presented by Pencina et al. (13). A positive NRI indicates that the UKPDS Risk Engine shows an improvement in classification over the Framingham equations, whereas a negative NRI corresponds to an improvement in classification of the Framingham equations over the UKPDS Risk Engine. The NRI for the diabetes group was 5.8% (P = 0.17), indicating that there was no significant improvement in classification using either equation. However, for those with nondiabetic hyperglycemia (NRI −14.0%, P = 0.004) and normoglycemia (NRI −12.4%, P < 0.001), the Framingham risk equations classified more participants correctly than the UKPDS Risk Engine.

Table 3

CVD risk classification comparing the UKPDS Risk Engine and Framingham risk equation models, including the NRI, for each population subgroup, EPIC-Norfolk cohort, U.K., 1993–2007

UKPDS risk categories	Framingham risk categories			Total
	0–<10%	10–<20%	≥20%
Participants with diabetes
0–<10%	17 (89.5)	3 (5.9)	0 (0.0)	20 (7.4)
10–<20%	2 (10.5)	34 (66.7)	9 (4.5)	45 (16.5)
≥20%	0 (0.0)	14 (27.5)	193 (95.5)	207 (76.1)
NRI (%), P value comparing UKPDS and Framingham models	5.8, 0.171
Participants with nondiabetic hyperglycemia
0–<10%	46 (26.1)	0 (0.0)	0 (0.0)	46 (5.1)
10–<20%	122 (69.3)	74 (24.3)	1 (0.2)	197 (21.7)
≥20%	8 (4.6)	230 (75.7)	425 (99.8)	663 (73.2)
NRI (%), P value comparing UKPDS and Framingham models	−14.0, 0.004
Participants with normoglycemia
0–<10%	2,177 (51.3)	10 (0.4)	0 (0.0)	2,187 (24.4)
10–<20%	2,002 (47.2)	972 (38.1)	36 (1.7)	3,010 (33.6)
≥20%	62 (1.5)	1,570 (61.5)	2,130 (98.3)	3,762 (42.0)
NRI (%), P value comparing UKPDS and Framingham models	−12.4%, < 0.001

The goodness-of-fit test statistics were nonsignificant for the UKPDS Risk Engine in the diabetes (P = 0.67) and hyperglycemia (P = 0.12) groups and for the Framingham equations in the hyperglycemia group (P = 0.12), indicating good calibration. However, the UKPDS Risk Engine was not well calibrated to the EPIC-Norfolk population in the normoglycemia group (P < 0.001) and the Framingham risk equations were not well calibrated in the diabetes (P = 0.02) and normoglycemia (P < 0.001) groups.

Results stratified by sex were broadly similar to the overall findings. In the diabetes group the overestimation of risk was not so pronounced in women, but there remained no significant difference in the ability of the two sets of risk equations to discriminate between those at high risk; NRIs for reclassification were nonsignificant in both sexes. Similarly, for hyperglycemic men, results were the same as the overall findings. In hyperglycemic women, however, the UKPDS Risk Engine was significantly better (P = 0.02) at discriminating between individuals at high risk, although the NRI was nonsignificant (P = 0.93). In normoglycemic women, the NRI became nonsignificant, indicating that that both sets of equations classified EPIC-Norfolk participants equally well. Conversely, in men, the Framingham equations was significantly better at discriminating between individuals at high risk and classified more participants correctly than the UKPDS Risk Engine (NRI −25%, P < 0.001).

Sensitivity analyses demonstrated that there were differences between participants with and without A1C data for certain baseline characteristics, e.g., individuals without A1C data were significantly older and had higher total cholesterol levels and systolic blood pressure. However, the absolute difference between the two groups did not affect the relative performance of the Framingham equations, which was the same in the two groups (data not shown).

CONCLUSIONS

In all population subgroups in the EPIC-Norfolk cohort, both the UKPDS Risk Engine and Framingham risk equations were moderately effective for identifying those at high risk (discrimination). If the purpose of a risk estimate is to rank individuals according to absolute risk to target therapy to those at greatest risk, then our results confirm that CVD risk equations can assist with the targeting of therapy in individuals with diabetes.

In addition to their ranking function, risk equations can be used to communicate prognostic information or accurate estimation of the likely absolute benefit from a therapeutic intervention to patients and practitioners. In this instance, the precise computation of absolute risk is important. Because both equations overestimated the risk of CVD in all subgroups, our results suggest that care is still needed when equations are being used to communicate risk information. Although the risk of a CVD event was overestimated using both risk tools, it was encouraging that the proportion of participants with true-positive results correctly identified (sensitivity) was high in all three subgroups. The Framingham equations performed better in the hyperglycemic and normoglycemic groups as they did not overestimate risk by as much as the UKPDS Risk Engine, and they classified more participants correctly.

These results are unsurprising because the UKPDS Risk Engine was developed specifically for use in those with diabetes, and its use to estimate risk in other populations was exploratory (14). Similarly, the overestimation of the Framingham risk equations confirms previous findings in populations with low disease rates (5). However, the definition of CVD used in this analysis contains fewer end points than the definition given by the Framingham equations, accounting for some of the overestimation.

The predictive ability of both the UKPDS and Framingham risk equations in the EPIC-Norfolk cohort is lower than that reported in the original populations in which they were developed (11,14). This is to be expected given changes in the nature and distribution of cardiovascular risk factors over time, both within and between populations. A systematic review of 27 external validity studies showed that the performance of the Framingham risk equations varies considerably among different countries and ethnic groups. Predicted-to-observed ratios ranged from an underprediction of 0.43 in a higher-risk population, to overprediction of 2.87 in lower-risk populations (5). Results from diabetic populations indicate that the Framingham equations underestimates risk by as much as a half (15–18), contrasting with results from this study in which Framingham equations overestimated CVD risk in individuals with diabetes. This finding may reflect the moderate CVD rates in the relatively healthy Norfolk region. Our results on the discrimination of the Framingham equations in the diabetic subgroup (aROC 0.73) are higher than those of a similar study, which validated the Framingham equations in a cohort of individuals with newly diagnosed diabetes (aROC 0.67) (18).

There are few diabetes-specific CVD risk equations available and a large number of equations for the general population. Whether the latter can be used in a subgroup of individuals with diabetes remains uncertain. The UKPDS Risk Engine was the first coronary risk calculator to be developed from a cohort with type 2 diabetes (14). Although it showed good predictive ability, individuals from the original study were not wholly representative of the general population. The authors advised that calculation of CVD risk in individuals who do not have newly diagnosed type 2 diabetes or who are aged <25 or >65 years should be completed with caution. The UKPDS equations have since been updated for use among individuals with established type 2 diabetes (version 3) (7), and the Risk Engine has been externally validated using data from the CARDS study (3,8). However, because the characteristics of the CARDS population are similar to that of the UKPDS, caution should still be used when calculating CVD risk in individuals outside the 25- to 65-year age range. The moderate predictive value of the UKPDS equation in the EPIC-Norfolk cohort can perhaps be attributed in part to the sizeable proportion of individuals aged >65 years (25%) in this cohort. As statin use was not common in the EPIC-Norfolk cohort at baseline, this is unlikely to account for the overestimation in risk using the UKPDS equation.

Measurement error in determining cardiovascular disease outcomes may have been present in our analyses. Four-fifths of the CVD events were nonfatal and were identified by linking records with hospital admission data. Although we could ascertain all deaths in the EPIC-Norfolk cohort, we could not identify all nonfatal cardiovascular events. However, previous validation studies in our cohort indicate high specificity of such case ascertainment (10). Hospital admission data probably underestimate nonfatal CVD events because not all of them result in hospital admission. Nevertheless, this method probably identifies nonfatal events of most clinical importance, e.g., those resulting in hospital admission.

The Framingham and EPIC-Norfolk CVD definitions included angina as an outcome, whereas the UKPDS definition did not. However, the CVD outcomes were largely similar, and this is unlikely to be a large source of bias. In terms of calculating the UKPDS risk equation, we did not have data on atrial fibrillation in the EPIC-Norfolk cohort. Because the number of participants with atrial fibrillation was very low in the UKPDS cohort (∼1%), the presence of atrial fibrillation is unlikely to affect our findings.

EPIC-Norfolk is a predominantly Caucasian cohort, which limits the generalizability of our findings on the performance of the two equations to other ethnic groups. In addition, both equations are based on information that might not be readily available in less developed health care settings, and the equations may need to be modified accordingly. Despite the large number of participants in EPIC-Norfolk, there was a low prevalence of individuals with diabetes at baseline (3%). This fact may have limited our ability to fully evaluate the predictive value of the risk estimates in this group, and further testing in other cohorts is recommended. The EPIC-Norfolk cohort may also have included a small proportion of individuals with type 1 diabetes. However, the number of participants receiving insulin therapy was low, indicating that this was unlikely to affect the overall findings. It is also possible that the nondiabetic hyperglycemic and normoglycemic groups contained some individuals with prevalent but undiagnosed diabetes. However, the UKPDS Risk Engine is used to estimate CVD in those with clinically diagnosed diabetes, so this is unlikely to be a major source of bias.

In this large, population-based cohort, we found that the UKPDS Risk Engine and Framingham risk equations performed reasonably well for identifying those with a high CVD risk (discrimination). However, both equations overestimated risk. Although CVD risk estimates may have a function in ranking individuals to target therapy to those at greatest risk, using equations to communicate absolute risk information needs careful consideration. The Framingham risk equations should continue to be used in the general population as 1) the equations did not overestimate risk by as much as the UKPDS Risk Engine in the normoglycemia and hyperglycemia groups and 2) they classified more participants correctly than the UKPDS Risk Engine, which is pertinent for statin prescribing. Further testing of the UKPDS (version 3) Risk Engine in other diabetic cohorts is required before it can replace Framingham-based methods of risk assessment in this group. It is clear that uncritical application of risk estimates may mislead patients and practitioners (19). It may therefore be valuable to focus on making sure that the tools we currently have for risk prediction are applied more broadly and routinely throughout clinical practice to address the gap between the promise of CVD prevention and its reality (20). In an attempt to reduce CVD risk, the precision of the instrument and how it is used can be considered of equal importance. Thus, there is still a need for further research into provider and patient perceptions of CVD risk (21,22) and the impact of knowledge of risk on behaviors.