Cardiovascular risk assessment scores for people with diabetes: a systematic review.

People with type 2 diabetes have an increased risk of cardiovascular disease (CVD). Multivariate cardiovascular risk scores have been used in many countries to identify individuals who are at high risk of CVD. These risk scores include those originally developed in individuals with diabetes and those developed in a general population. This article reviews the published evidence for the performance of CVD risk scores in diabetic patients by: (1) examining the overall rationale for using risk scores; (2) systematically reviewing the literature on available scores; and (3) exploring methodological issues surrounding the development, validation and comparison of risk scores. The predictive performance of cardiovascular risk scores varies substantially between different populations. There is little evidence to suggest that risk scores developed in individuals with diabetes estimate cardiovascular risk more accurately than those developed in the general population. The inconsistency in the methods used in evaluation studies makes it difficult to compare and summarise the predictive ability of risk scores. Overall, CVD risk scores rank individuals reasonably accurately and are therefore useful in the management of diabetes with regard to targeting therapy to patients at highest risk. However, due to the uncertainty in estimation of true risk, care is needed when using scores to communicate absolute CVD risk to individuals.

Introduction

Type 2 diabetes is a strong risk factor for cardiovascular disease (CVD), with some studies suggesting that it confers an equivalent risk to having had a myocardial infarction [1, 2]. Multifactorial interventions, such as the Steno-2 study, have been effective in reducing the risk of non-fatal [3] and fatal CVD [4] among diabetic patients through therapy targeting hyperglycaemia, hypertension and hypercholesterolaemia. Despite this evidence of effectiveness, many countries use a rationing approach to the prescription of cardiovascular risk reduction treatment, with national guidelines suggesting that patients should have their risk of CVD calculated, to ensure therapy is targeted to patients at highest absolute risk. Multivariate risk scores have, therefore, been used to predict CVD risk in individuals with diabetes.

There are a large number of scores for the general population, but few that are specific to people with diabetes. Whether general population scores can accurately be used in a subgroup of individuals with diabetes is unclear [5, 6]. The most commonly used score is that originally developed in 5,573 men and women participating in the Framingham study in the early 1970s, which, in general, performs well in North America [7], but less well in other populations [7-9]. We aimed to review the published evidence on performance of CVD risk scores in diabetic populations. First, we examined the overall rationale for using cardiovascular risk scores in patients with diabetes. Second, we provide results from a systematic review of the published literature on CVD risk scores that have been developed or evaluated in individuals with diabetes. Finally, we explored methodological issues surrounding the development, validation and comparison of risk scores.

Why estimate CVD risk in individuals with diabetes?

There are several reasons why it may be important to quantify the risk of developing CVD in patients with diabetes. The clear identification of the rationale for developing a risk score is critical to how its validity is assessed. For example, if the purpose of a score is to rank individuals and groups according to absolute risk for the purpose of targeting therapy to those at greatest risk, then it is the ranking that is important and not necessarily the absolute risk estimate. If, on the other hand, the principal justification is to provide prognostic information or accurate estimation of the likely absolute benefit from a therapeutic intervention, then a precise computation of absolute risk is important. Finally, if the main reason for calculating risk as part of a preventive strategy is to motivate patients to change their behaviour and adhere to medical treatments, it may be important to calculate modifiable risk rather than use a score dominated by fixed variables. In the following sections we describe the origin of CVD risk scores and illustrate how purpose, construction and validation are often disconnected.

CVD risk scores have been widely used in the UK and elsewhere for over 10 years and were introduced at a time when the cost of cholesterol-lowering drugs was an important political issue [10]. National recommendations began appearing in the mid-1990s, when specified annual CVD risk estimates were used to determine thresholds for prescribing therapy. These thresholds were decided on the basis of the number needed to treat to prevent one CHD event, the cost-effectiveness of treatment and, predominantly, the proportion of the population requiring treatment and the total cost of treatment [11]. As such, the threshold for intervention based on the ranking of absolute risk was largely a financial decision. Although results from the Heart Protection Study [12] and the Collaborative Atorvastatin Diabetes Study [13] suggest that statin therapy is effective in patients with diabetes at high CVD risk, irrespective of their initial cholesterol concentrations, not all health systems can afford such a policy. The ranking of absolute risk is clearly important for making collective decisions about therapy, but, beyond statin prescribing, does the calculation of CVD risk aid clinical decision-making?

An estimate of absolute risk can inform the potential for absolute risk reduction, providing patients with an idea of expected benefit from a therapy or intervention. This is important for individual, rather than collective, decision-making. However, further research is needed to understand the process by which the clinician and patient interact once cardiovascular risk has been assessed [9]. While it is assumed that telling individuals their CVD risk is a motivating tool, there is only weak evidence that this assumption is valid. In a systematic review of randomised controlled trials that evaluated the effectiveness of risk-scoring methods, no strong evidence was found that a CVD risk assessment by a clinician improved CVD-related health outcomes [9]. Practitioners need to provide information about what patients can do to reduce this risk; they also need to acknowledge the possibility of false reassurance or a fatalistic response, both of which can in theory lead to an increase in population risk. Qualitative work has demonstrated that very few patients with type 2 diabetes understand the direct link between having diabetes and their CVD risk [14]. Although they are aware of CVD, they are more likely to attribute it to external or immutable factors such as stress and heredity, rather than modifiable risk factors such as high cholesterol, hyperglycaemia and smoking [14]. This phenomenon links in with the idea of using risk scores as preventive or motivating tools. Most of the predictive value of a CVD risk score comes from the inclusion of two unchangeable risk factors, age and sex. It is probably difficult to persuade patients to change their behaviour on the basis of CVD risk scores that are mostly driven by risk factors patients cannot change. Thus risk scores that only incorporate modifiable risk factors are more likely to be useful for preventive strategies in patients with diabetes.

Systematic review of CVD risk assessment tools

Search strategy

A comprehensive literature search for studies of CVD risk assessment tools was performed using MEDLINE, Web of Science and Cochrane Reviews from database inception up to 30 June 2008. The search strategy focused on four key elements: CVD, type 2 diabetes, risk assessment/score/prediction and specific names of known risk scores (see Appendix). We also screened the reference lists of papers identified from the initial electronic search. No language restriction was applied; articles were translated when necessary.

Selection criteria

We included studies reporting CVD risk assessment tools or scores that: (1) were derived from prospective cohort studies or randomised trials; (2) were derived in the general population and evaluated in individuals with diabetes, or developed in a diabetic population; and (3) reported a measure of performance of the risk score for predicting CVD. We restricted the review to studies that reported the following diseases as a primary outcome:

Fatal or non-fatal CVD

Fatal or non-fatal CHD

Fatal or non-fatal cerebrovascular disease or stroke

We excluded studies that derived cardiovascular risk scores for the general population but did not evaluate them in individuals with diabetes. We excluded studies that derived risk prediction tools other than score-type tools, such as those using carotid ultrasonography and myocardial perfusion scintigraphy. If scores and their evaluation were reported in several different papers, we included the score only once by selecting the paper that reported the most information on predictive ability.

Data extraction

Two reviewers (P. Chamnan, R. K. Simmons) independently reviewed the results from the primary search of titles, followed by the abstract and full paper searches (Fig. 1). Where reviewers disagreed, consensus was reached through discussion. The two reviewers used a standardised form to extract data on the performance of the risk scores. This included the name of the risk score and study, the country and setting, details on derivation and validation populations, follow-up for derivation and validation cohorts, definition of diabetes and CVD, risk factors included in the scores, and measures of predictive ability, including discrimination, calibration, sensitivity, specificity, and positive and negative predictive value. We also extracted data from original studies if articles identified through the initial search did not contain information on the development or validation of risk scores (Tables 1 and 2).

Fig. 1

Flow of identification of included studies

Table 1

Summary of the derivation of CVD/CHD risk scores primarily developed in individuals with diabetes

Name of risk score	Derivation population (setting)	Definition of diabetes	Follow-up time	Main outcome	Risk factors included in score
UKPDS risk engine (UKPDS 56) [47]	4,540 men and women aged 25–65 years with diabetes, without history of MI, angina or heart failure (UK)	Newly physician-diagnosed diabetes and FPG >6 mmol/l on two further occasions	Median F/U 10.3 years	Fatal or non-fatal MI, or sudden death (verified by two independent clinical assessors)	Age at diagnosis of DM, sex, ethnicity, smoking, HbA1c, systolic BP, total-cholesterol:HDL-cholesterol ratio, duration of DM
Oxford risk engine (UKPDS risk engine version 3) [48]	3,465 men and women aged 25–65 years with and without diabetes with no known CVD (UK)	Newly physician-diagnosed diabetes and FPG >6 mmol/l on two further occasions	Median F/U 5 years	CVD (first occurrence of fatal or non-fatal MI, sudden cardiac death, other ischaemic heart disease, fatal or non-fatal stroke, or fatal peripheral vascular disease)	Age at diagnosis of DM, sex, ethnicity, smoking, HbA1c, systolic BP, total cholesterol:HDL-cholesterol ratio, duration of DM
Diabetes Audit and Research in Tayside, Scotland (DARTS) [17]	4,569 men and women of any age with diabetes without previous cardiovascular events (Scotland)	Those at any age treated with diet or oral hypoglycaemic agents or those aged >35 years with diagnosis of diabetes (Web-based information system) (Sensitivity/specificity of the ascertainment 97%)	Mean (SD) F/U 4.7 (6.1) years	First major CHD event (fatal or non-fatal acute MI or CHD death)	Age at diagnosis of DM, sex, systolic BP, duration of DM, smoking, total cholesterol, HbA1c, treated hypertension, height
Swedish National Diabetes Register (NDR) [23]	5,823 men and women aged 18–70 years with diabetes and no previous CVD (Sweden)	Diagnosis of diabetes from the Swedish National Diabetes Register	Mean F/U 5.6 years	Fatal or non-fatal CVD (CHD or stroke, including coronary intervention) retrieved by data linkage	Age at diagnosis of DM, sex, smoking, duration of diabetes, BMI, systolic BP, HbA1c, antihypertensive drugs, lipid-lowering drugs
Atherosclerosis Risk in Communities (ARIC) [15]	1,500 men and women aged 45–64 years with diabetes, initially free of CHD (USA) (no validation)	FPG >7.0 mmol/l, non-fasting glucose >11.1 mmol/l, self-reported physician-diagnosed diabetes or pharmacological treatment for diabetes	Median F/U 10.2 years	CHD (validated definite or probable hospitalised MI, definite CHD death, an unrecognised MI defined by ECG readings, or coronary revascularisation)	Age, ethnicity, smoking, systolic BP, use of antihypertensive medication, total cholesterol, HDL; alternative model includes: BMI, WHR, sport activity, Keys score for diet, serum creatinine, serum albumin, WBC, factor VIII, LVH, and IMT
Hong Kong Diabetes Registry [18]	3,521 Chinese men and women with median age of 57 years without a history of CHD or heart failure (Hong Kong)	Type 2 diabetes referred from GP and special clinics, and patients discharged from hospitals	Median F/U 5.59 years	Total CHD (MI or IHD) retrieved from discharge diagnosis	Age, sex, smoking, duration of diabetes, systolic BP, GFR, ACR, non-HDL-cholesterol
UKPDS for stroke (UKPDS 60) [49]	4,549 men and women aged 25–65 years with diabetes, no known stroke (UK)	Newly physician-diagnosed diabetes, and FPG >6 mmol/l on two further occasions	Median F/U 10.5 years	Stroke defined as neurological deficit with symptoms or signs lasting 1 month or more	Age at diagnosis of DM, sex, smoking, atrial fibrillation, systolic BP, total:HDL-cholesterol ratio, duration of DM
Hong Kong Diabetes Registry for stroke [26]	3,668 diabetic patients without history of stroke, referred to the Prince of Wales Hospital (Hong Kong)	Type 2 diabetes referred from GP and special clinics, and patients discharged from hospitals	Median F/U 5.4 years	Fatal and non-fatal haemorrhagic and ischaemic stroke (data retrieved via the Hong Kong hospital authority central computer system)	Age, HbA1c, ACR, history of CHD

ACR, albumin:creatinine ratio; DM, type 2 diabetes mellitus; F/U, follow-up; IHD, ischaemic heart disease; IMT, intima-medial thickness; LVH, left ventricular hypertrophy; MI, myocardial infarction; WBC, white blood cell count

Table 2

Summary of the derivation of CVD/CHD risk scores primarily developed in the general population

Name of risk score	Study population (setting)	Definition of diabetes	Follow-up time	Main outcome	Risk factors included in score
Framingham 1991 [50]	5,573 US white men and women aged 30–74 years with no overt CHD (6% DM)	Treatment with insulin or oral hypoglycaemic drugs, blood glucose >8.3 mmol/l at two clinic visits, or fasting blood glucose >7.8 mmol/l at initial examination of the Offspring Study [51]	12 years	Myocardial infarction, CHD, death from CHD, stroke, CVD, death from CVD	Age, sex, blood pressure, total cholesterol, HDL-cholesterol, smoking, glucose intolerance, left ventricular hypertrophy by echocardiogram
Framingham 1998 [52]	5,345 US white men and women aged 30–74 years with no overt CHD (6% DM)	Treatment with insulin or oral hypoglycaemic drugs, blood glucose >8.3 mmol/l at two clinic visits, or fasting blood glucose >7.8 mmol/l at initial examination of the Offspring Study [51]	12 years	CHD (angina pectoris, recognised and unrecognised MI, coronary insufficiency, CHD death) Hard CHD included total CHD without angina pectoris	Age, sex, diabetes, smoking, blood pressure, total or LDL-cholesterol, HDL-cholesterol
PROCAM score [53]	5,389 German white working men aged 35–65 years with no history of MI, stroke, angina pectoris or baseline ECG signs of IHD (Germany) (16.1% DM)	N/A	10 years	Major coronary event (sudden cardiac death, definite fatal or non-fatal MI on basis of ECG and/or cardiac enzyme changes)	Age, smoking, diabetes, family history of premature MI, systolic blood pressure, HDL-cholesterol, LDL-cholesterol, triacylglycerol
Joint British Societies Risk Chart (JBSRC) [54]	Computerised CHD risk prediction chart based on Framingham risk equations	Not applicable	Not applicable	Coronary heart disease	Age, sex, systolic and diastolic BP, total and HDL-cholesterol, smoking, diabetes, presence of left ventricular hypertrophy from ECG
CardioRisk Manager calculator (CRM), computerised version of Framingham equation [55]	Computerised version of full equations of Framingham risk score to test against a placebo arm of West of Scotland Coronary Prevention Study (WOSCOPS) [56]	Not applicable	Not applicable	CHD or stroke	CHD: age, sex, blood pressure, total cholesterol, HDL-cholesterol, smoking, glucose intolerance, left ventricular hypertrophy by echocardiogram; stroke: age, systolic blood pressure, diabetes, smoking, prior CVD, atrial fibrillation, left ventricular hypertrophy by electrocardiogram, use of antihypertensive medication prior to occurrence of stroke
Myocardial Infarction Population Registry of Girona (REGICOR) [57]	Men and women aged 35–74 in Girona, Spain (data from myocardial infarction population registry of Girona, REGICOR)	N/A	10 years	Fatal or non-fatal AMI, with or without symptoms, or angina	Age, sex, total cholesterol, HDL-cholesterol, blood pressure, smoking, diabetes
SCORE [16]	205,178 European men and women aged 19–80 years with no history of MI (12 European countries)	Not applicable	Varies among cohorts, up to 25 years	CVD mortality (ICD-9 codes 401–414 and 426–443, with the exception of non-atherosclerotic causes of death: 426.7, 429.0, 430.0, 432.1, 437.3, 437.4 and 437.5; ICD-9 codes 798.1 and 798.2 were also classified as CVD deaths)	Age, sex, systolic blood pressure, smoking, total cholesterol or total cholesterol:HDL-cholesterol ratio
DECODE score [58]	25,413 European men and women aged 30–74 years (14 European studies) (4% known DM, 5% screen-detected DM )	Known diabetes and screen-detected diabetes using OGTT	4.8 to more than 10 years	CVD mortality (CVD codes 401–448 of the 8th and 9th ICD revisions, code I10–I79 of the 10th revision)	Age, sex, FPG and 2-h plasma glucose, diabetes, smoking, systolic blood pressure, total cholesterol
Framingham Stroke [59]	5,734 men and women aged 55–84 years with no history of stroke (USA)	Treatment with insulin or oral hypoglycaemic drugs, and fasting blood glucose	10 years	Brain infarction and transient ischaemic attack, cerebral embolus, intracranial haemorrhage	Age, systolic blood pressure, diabetes, smoking, prior CVD, atrial fibrillation, left ventricular hypertrophy by electrocardiogram, use of antihypertensive medication prior to occurrence of stroke

AMI, acute myocardial infarction; DM, type 2 diabetes mellitus; ICD, international classification of disease; IHD, ischaemic heart disease; MI, myocardial infarction; N/A, not available

Results

Our electronic search retrieved 2,113 potentially relevant papers (Fig. 1). After reviewing titles, abstracts, full texts and citation lists, 13 articles reporting the predictive performance of 17 different CVD risk scores met the inclusion criteria. We provide a summary of the derivation of the risk scores in Tables 1 and 2, and a summary describing the evaluation of the scores in Table 3. One paper was translated.

Table 3

Performance of CVD risk scores evaluated in individuals with diabetes

Name of risk score	Validation population	Definition of diabetes	Outcome (n events)	Discrimination	Calibration	Sensitivity and specificity (%)	PPV/NPV (%)
UKPDS risk engine [47]	428 men and women aged 30–74 years with newly diagnosed diabetes without pre-existing CVD, mean F/U 4.2 (0.6) years (GP recruitment in Poole, UK) [21]	Surveillance programme to identify diabetes cases; methods of diagnosis, N/A	CHD determined via hospital and GP notes, death certificates and post-mortem examination record (98)	CHD: aROC 0.670 (0.598–0.742)	Poor calibration for CHD: HL χ2 17.1 (p = 0.029)	10 year CHD risk >15%: Sen 89.8 (95% CI 82.0–95.0) Spc 30.3 (95% CI 25.4–35.6)	N/A
	339 diabetic patients with 10 years of follow-up (Greece) [25]	N/A	CHD determined by coronary angiography (108)	aROC 0.61	N/A	Sen 56; Spc 56 (no data on which risk threshold was used)	PPV 37; NPV 73 (no data on which risk threshold was used)
	112 patients with type 2 diabetes with 10 years of follow-up (Spain) [22]	1985 WHO criteria (FPG >7.7 mmol/l or, 2 h OGTT >11.0 mmol/l)	CHD event (19)	N/A	Men: overestimation by 108.8%; women: overestimation by 51.3%	10 year CHD risk >20%: men: Sen 67; Spc 7; women: Sen 71; Spc 59	10 year CHD risk >20%: men: PPV 14; NPV 50; women: PPV 24; NPV 92
	798 diabetic patients identified via a computerised clinic database, follow-up 10 years [19]	N/A	CHD and CVD determined through computerised clinical records (358 CVD, including 269 CHD events)	CVD: c = 0.74; CHD: c = 0.65	Poor calibration (p < 0.001); observed:predicted events ratio 1.20 and 1.60 for CVD and CHD (r = 0.89 and 0.86 respectively)	N/A	N/A
	3,546 Chinese men and women with median age of 56 years without a history of CHD or HF, median F/U of 5.59 years (Hong Kong) [18]	Type 2 diabetes referred from GP and special clinics, and those discharged from hospitals	Hard CHD: myocardial infarction and CHD death (170)	Overall c statistic 0.610 (95% CI 0.581–0.639)	Poor calibration: HL χ2 = 230.1; p < 0.0001	N/A	N/A
Oxford risk engine (UKPDS risk engine version 3) [48]	1,410 men and women aged 40–75 years on the placebo arm of the Collaborative Atorvastatin Diabetes Study (CARDS) (median F/U 3.9 years) [48]	1985 WHO criteria (computerised registers of patients and by opportunistic assessment of individuals attending diabetes clinics)	CVD event (189)	N/A	Underestimation by 10.6%; 189 observed events vs 169 predicted events over 3.9 years	N/A	N/A
Diabetes Audit and Research in Tayside, Scotland (DARTS) [17]	Salford Diabetes Information System, F/U 5 years [17]	Treatment with diet or oral hypoglycaemic agents/age >35 years	CHD determined via hospital episode statistics (N/A)	c statistic = 0.69 (95% CI 0.58–0.78)	Included a graph but no statistical test	N/A	N/A
Swedish National Diabetes Register (NDR) [23]	5,823 men and women aged 18–70 years with diabetes, no previous CVD, F/U 5.6 years (Swedish) [23]	Diagnosis of diabetes from the Swedish National Diabetes Register	Fatal and non-fatal CVD (N/A)	c statistic = 0.69	Good calibration; observed:predicted CVD rate ratio 0.998	N/A	N/A
	3,068 men and women aged 18–70 years with diabetes, no previous CVD, F/U 4 years (Swedish) [23]	Diagnosis of diabetes from the Swedish National Diabetes Register	Fatal and non-fatal CVD (261)	c statistic = 0.69	Good calibration; observed:predicted CVD rate ratio 0.96	N/A	N/A
Hong Kong Diabetes Registry for CHD [18]	3,546 Chinese men and women with median age of 56 years without CHD or HF, median follow-up of 5.59 years (Hong Kong) [18]	Type 2 diabetes referred from GP and special clinics, and those discharged from hospitals	CHD: myocardial infarction or ischaemic heart disease (170)	Overall aROC 0.704 (95% CI 0.675–0.733) Adjusted aROC 0.737	Good calibration: HL χ2 = 14.05; p > 0.05	5 year CHD risk >5.2%: Sen 67.6; Spc 68.5	5 year CHD risk >5.2%: PPV 8.6; NPV 98.0
Framingham 1991 [50]	3,898 men and women aged 25–65 years with diabetes without history of MI, angina or stroke (UKPDS) [24]	Newly physician-diagnosed diabetes, and FPG >6 mmol/l on two further occasions	Fatal CVD and fatal CHD (CVD: 7.4%, CHD: 6.3%)	aROC 0.76	32% underestimation	N/A	N/A
	112 patients with type 2 diabetes with 10 years of follow-up (Spain) [22]	1985 WHO criteria (FPG >7.7 mmol/l or 2 h OGTT >11.0 mmol/l)	CHD event (19)	N/A	Men: overestimation by 64.8%; women: overestimation by 31.6%	10 year CHD risk >20%: men: Sen 75; Spc 17; women: Sen 71; Spc 67	10 year CHD risk >20%: men: PPV 17; NPV 75; women: PPV 28; NPV 93
	428 men and women aged 30–74 years with newly diagnosed diabetes without pre-existing CVD, mean F/U of 4.2 (0.6) years (GP recruitment in Poole, UK) [21]	Surveillance programme to identify diabetes cases, methods of diagnosis, N/A	CVD and CHD determined via hospital and GP notes, death certificates and post-mortem examination record (98)	CVD: aROC 0.673 (0.612–0.734); CHD: aROC 0.657 (0.581–0.732)	Poor calibration for CVD and CHD CVD HL χ2 = 32.8 (p < 0.001); CHD HL χ2 = 19.8 (p = 0.011)	10 year CHD risk >15%: Sen 85.7 (95% CI 77.8–99.5); Spc 33.0 (95% CI 30.7–34.7)	N/A
	938 men and women with type 1 and type 2 diabetes, F/U 4 years (Cardiff, UK) [20]	Type 1 and type 2 diabetes identified through the Cardiff Diabetes Database; methods of diagnosis, N/A	CHD via primary or subsidiary diagnosis in hospital record linkage (172)	Men: aROC 0.64; women: aROC 0.66	N/A	10 year CHD risk >30%: men: Sen 34%, women: Sen25%; 10 year CHD risk >20%: men: Sen 75%; women: Sen 58%	10 year CHD risk >30%: men: PPV 26%, NPV 83%; women: PPV 33%, NPV 86%; 10 year CHD risk >20%: men: PPV 24%, NPV 88%; women: PPV 24%, NPV 88%
Framingham 1998 [52]	112 patients with type 2 diabetes with 10 years of follow-up (Spain) [22]	1985 WHO criteria (FPG >7.7 mmol/l or 2 h OGTT >11.0 mmol/l)	CHD event (19)	N/A	Men: overestimation by 97.8%; women: overestimation by 38.2%	10 year CHD risk >20%: men: Sen 75; Spc 11; women: Sen 71; Spc 69	10 year CHD risk >20%: men: PPV 16; NPV 67; women: PPV 29; NPV 93
	339 diabetic patients, follow-up of 10 years (Greece) [25]	N/A	CHD determined using coronary angiography (108)	aROC 0.65	N/A	Sen 55; Spc 65 (no data on which risk threshold was used)	PPV 43; NPV 75 (no data on which risk threshold was used)
PROCAM score [53]	798 diabetic patients identified via computerised clinic database with 10 years of follow-up (UK) [19]	N/A	CHD and CVD determined through computerised clinical records (358 CVD, including 269 CHD events)	CVD: c = 0.67; CHD: c = 0.76	Poor calibration (p < 0.001); observed:predicted events ratio 2.79 and 2.05 for CVD and CHD (r = 0.79 and 0.81 respectively)	N/A	N/A
Joint British Societies Risk Chart (JBSRC) [54]	798 diabetic patients identified via computerised clinic database with 10 years of follow-up (UK) [19]	N/A	CHD and CVD determined through computerised clinical records (358 CVD, including 269 CHD events)	CVD: c = 0.80; CHD: c = 0.77	Poor calibration (p < 0.001)	N/A	At CVD risk >30%: PPV 85%; at CHD risk >30%: PPV 66%
CardioRisk Manager calculator (CRM) [55]	798 diabetic patients identified via a computerised clinic database with 10 years of follow-up (UK) [19]	N/A	CHD and CVD determined through computerised clinical records (358 CVD, including 269 CHD events)	CVD: c = 0.76; CHD: c = 0.73	Poor calibration (p < 0.001) observed:predicted events ratio 2.30 and 1.74 for CVD and CHD (r = 0.98 and 0.97 respectively)	N/A	N/A
Myocardial Infarction Population Registry of Girona (REGICOR) [57]	112 patients with type 2 diabetes with 10 years of follow-up (Spain) [22]	1985 WHO criteria (FPG >7.7 mmol/l or 2 h OGTT >11.0 mmol/l)	CHD event (19)	N/A	Men: underestimation by 17.6%; women: underestimation by 34.2%	10 year CHD risk >20%: men: Sen 25; Spc 67; women: Sen 0; Spc 100	10 year CHD risk >20%: men: PPV 14; NPV 80; women: PPV –; NPV 85
SCORE [16]	3,898 men and women aged 25–65 years with diabetes, without history of MI, angina or stroke within 1 year (UKPDS) [24]	Newly physician-diagnosed diabetes and FPG >6 mmol/l on two further occasions	Fatal CVD and fatal CHD (CVD: 7.4%, CHD: 6.3%)	aROC 0.77	18% overestimation	N/A	N/A
DECODE score [58]	3,898 men and women aged 25–65 years with diabetes, without history of MI, angina or stroke within 1 year (UKPDS) [24]	Newly physician-diagnosed diabetes and FPG >6 mmol/l on two further occasions	Fatal CVD (CVD: 7.4%, CHD: 6.3%)	aROC 0.67	11% underestimation	N/A	N/A
UKPDS for stroke [49]	1,370 patients aged >30 years of the Wisconsin Epidemiologic Study of Diabetic Retinopathy cohort, follow-up 8.3 years (USA) [49]	N/A	Fatal stroke (197)	N/A	Underestimation of fatal stroke by 12.5%	N/A	N/A
	3,541 Chinese diabetic patients without history of stroke (Hong Kong) [26]	Physician-diagnosed diabetes (GP and special clinics, and those discharged from hospital)	Stroke: hospital discharge diagnosis confirmed by physicians (182)	Unadjusted aROC 0.588 (95% CI 0.549–0.626)	N/A	N/A	N/A
Hong Kong Diabetes Registry for stroke [26]	3,541 Chinese diabetic patients without history of stroke, median follow-up of 5.37 years (Hong Kong) [26]	Physician-diagnosed diabetes referred from GP and special clinics, and those discharged from hospital	Stroke: hospital discharge diagnosis confirmed by physicians (182)	Adjusted aROC 0.770 for haemorrhagic stroke; 0.785 for ischaemic stroke	N/A	5 year stroke risk >6.1%: Sen 65.7; Spc 74.9	5 year stroke risk >6.1%: PPV 11.6
Framingham stroke risk [59]	178 men and women with diabetes, with or without metabolic syndrome (Spain) [60]	FPG and OGTT based on WHO criteria	Stroke (9)	N/A	Predicted 10 year risk varied from 17.3 to 19.1%; observed 5 year cumulative incidence varied from 3.5 to 5.8	N/A	N/A

F/U, follow-up; HF, heart failure; HL χ2, Hosmer–Lemeshow χ2; MI, myocardial infarction; N/A, not available; NPV, negative predictive value; PPV, positive predictive value; Sen, sensitivity; Spc, specificity

Development of risk scores

Out of 17 different risk scores, 15 were developed in predominantly white populations (USA and Europe) and two were developed in Chinese populations (Hong Kong). Cohort size ranged from 1,500 [15] to 205,178 individuals [16] and follow-up time from 4.7 [17] to 25 years [16]. The age of men and women included in development cohorts ranged from 18 to 84 years; patients with previous CVD were usually excluded. Eight risk scores were originally developed in a cohort of individuals with diabetes (Table 1), while the other nine were developed in a general population and subsequently evaluated in a cohort of individuals with diabetes (Table 2). The majority of risk scores (n = 10) provided estimates of risk for CHD. Two risk scores estimated risk of non-fatal or fatal CVD outcomes, while two risk scores predicted CVD deaths. The remaining three scores estimated risk of non-fatal or fatal stroke. The majority of risk scores incorporated classic CVD risk factors, such as age, sex, smoking, blood pressure and total cholesterol. Most risk scores derived from the general population contained a dichotomous variable for diabetes (yes/no) (n = 8) and did not take account of diabetes-specific risk factors, such as duration of diabetes or glycaemia. Conversely, risk scores developed in individuals with diabetes often included age at diagnosis, duration of diabetes and/or a measure of glycaemic control, such as HbA1c or fasting plasma glucose (FPG).

Evaluation of risk scores

Sixteen risk scores were evaluated in 13 different validation cohorts. Only the Atherosclerosis Risk in Communities risk score was not evaluated in an external validation cohort. The majority of validation cohorts were based in Europe (n = 10) and varied in size from 112 to 5,823 individuals with diabetes (Table 3). The 10-year cumulative incidence of CVD varied considerably from 5% in a Chinese cohort [18] to 45% in a British cohort [19]. The age of individuals ranged from 18 to 75 years at baseline and median follow-up time from 4 to 10 years. All studies validated risk scores in patients with type 2 diabetes, including one study that evaluated a risk score in type 1 and type 2 diabetes [20]. Individuals with diabetes were recruited from the general population (n = 7) or specialist diabetes clinics (n = 6) and were identified through (1) computerised databases or registries in five cohorts; (2) clinical records in seven cohorts; and (3) from the placebo arm of a trial in one cohort. Only three studies included a clear definition of diabetes, e.g. whether individuals were diagnosed by FPG or an OGTT according to WHO criteria (Table 3). Two articles failed to include any information on how individuals with diabetes were identified or diagnosed. Similarly, four articles did not include a clear definition of which CVD endpoint was used, with definitions varying between studies, e.g. clinical diagnosis of fatal and non-fatal CVD, CHD determined using coronary angiography or fatal CVD. Clinical records were used to retrieve data on CVD endpoints (n = 11) in most validation cohorts and these were confirmed by expert reviewers in two studies. Diagnosis of CVD was made using coronary angiography in one validation study in Greece (Table 3). The majority of validation studies compared the performance of different CVD risk scores in a single population (n = 7). Among these, four studies compared the predictive ability of risk scores developed in individuals with diabetes and those developed in a general population.

Performance of the risk scores

Table 3 summarises the measures of predictive ability assessed in each validation study. Few studies reported complete measures of predictive performance, including discrimination, calibration and global model fit. The majority of studies reported a measure of discrimination (area under the receiver operating characteristic curve [aROC], also known as a c-statistic). Risk scores predicting CVD or CHD outcomes showed moderate to good discriminatory power in validation cohorts (aROC range 0.61 to 0.80). Similarly, risk scores predicting stroke reported aROCs from 0.59 to 0.79.

Three risk scores were evaluated using a Hosmer–Lemeshow χ2 statistic as a measure of calibration in two validation cohorts, while eight studies reported either predicted and observed event rates, predicted to observed rate ratio or whether the scores over- or underestimated CVD risk. Two validation studies did not report any measure of calibration. The UK Prospective Diabetes Study (UKPDS) risk engine showed poor calibration in both British [19, 21] and non-British validation cohorts [18, 22], while the Swedish National Diabetes Register [23] and Hong Kong Diabetes Registry [18] reported good calibration in Swedish and Chinese populations respectively. Most risk scores developed in the general population underestimated CVD risk in diabetic patients. We found underestimations ranging from 11% for fatal CVD risk using the Diabetes Epidemiology: Collaborative Analysis of Diagnostic Criteria in Europe (DECODE) score [24] to 64% for fatal and non-fatal CVD risk using the Prospective Cardiovascular Münster (PROCAM) score [19]. Coleman et al. validated the Framingham, Systematic Coronary Risk Evaluation (SCORE) and DECODE risk engines in a population of 3,898 individuals with diabetes and showed that they did not provide reliable fatal CVD and CHD risk estimates [24]. Conversely, in populations with low CVD risk, such as Mediterranean countries, most of the risk scores overestimated CVD risk, even those developed in individuals without diabetes [22].

Studies comparing the predictive ability of a CVD risk score developed in a diabetic cohort and a score developed in the general population reported inconsistent results. In a study of 339 patients with diabetes, Protopsaltis et al. showed that the Framingham risk equations were more accurate than the UKPDS risk engine for predicting coronary artery disease risk (aROC 0.65 and 0.61 respectively) [25]. With similar sensitivity, Framingham risk equations had higher specificity and positive and negative predictive values than the UKPDS risk engine (65%, 43% and 75% vs 56%, 37% and 73%, respectively). However, the study population was small and the diagnosis of coronary artery disease was established by means of coronary angiography, which includes individuals with sub-clinical CHD, an endpoint not included in the UKPDS definition. By contrast, in a British community-based cohort of 428 individuals with newly diagnosed type 2 diabetes [21], the Framingham risk equations appeared to underestimate cardiovascular and coronary disease events by 33% and 32%. The UKPDS risk engine had lower levels of underestimation at 13%, suggesting that this diabetes-specific risk score performed better than a risk score developed in the general population. However, both the Framingham equations and UKPDS risk engine showed modest discriminatory ability and poor calibration (aROC of 0.66 and 0.67, Hosmer–Lemeshow χ2 of 19.8 [p = 0.011] and 17.1 [p = 0.029] respectively), making it difficult to reach any firm conclusions.

Sensitivity, specificity and both positive and negative predictive values varied considerably between risk scores and validation cohorts, although most studies reported moderate to good sensitivity and specificity (Table 2). Risk scores for CHD and stroke developed in the Hong Kong Diabetes Register had good sensitivity and specificity when tested in Chinese validation cohorts [18, 26], while the Framingham risk score and the UKPDS risk engine also had good sensitivity, but relatively low specificity when tested in European populations [21, 22].

Discussion

This systematic review has shown that the predictive ability of CVD risk scores, which were developed mainly for White populations, varies considerably between different populations. There is little evidence to suggest that using risk scores developed in individuals with diabetes will help to estimate CVD risk among diabetic patients more accurately than use of those developed in the general population. The inconsistency in methods used to evaluate CVD risk scores makes it difficult to compare or summarise the predictive ability of different risk scores.

Our review supports previous research showing that CVD risk scores developed in the general population are likely to underestimate CVD risk in individuals with diabetes [9]. In theory, one way of addressing this underestimation would be to use risk prediction tools solely derived from diabetic populations [10]. However, our results suggest that diabetes-specific risk engines need to be validated in other populations before they are widely adopted and can replace Framingham-based methods of risk assessment. The issue of predicting risk for people with diabetes is likely to be increasingly complex, as a greater proportion of patients are treated with CVD risk factor modifying therapy. The development of risk prediction scores in populations with long-standing and treated diabetes, possibly incorporating information about the degree of glycaemic control, would be an important issue for future research. At the same time, the prediction of absolute future CVD risk in a clinical situation is likely to be based on single baseline measures of cardiovascular risk factors and not on any time-averaged measures collected during repeat visits. While the latter has a role in assessing the aetiological association between risk factors and outcome, repeated measures have limited utility in the practical clinical situation of risk prediction.

Most diabetes-specific risk scores incorporate measures of glycaemic control such as FPG and HbA1c, while most scores developed in the general population include a binary variable indicating whether an individual has diabetes or not. Hence, an important question in the general population is whether adding measures of glycaemic control improves the predictive ability of existing risk scores incorporating traditional CVD risk factors. In populations of people with diabetes, studies comparing the predictive performance of the Framingham risk score and the UKPDS risk engine have shown conflicting results [21, 25]. Part of the uncertainty about the additional predictive contribution of hyperglycaemia in this context is explained by the fact that, in comparison with age, cholesterol and smoking, hyperglycaemia is a relatively weak CVD risk factor, thus making it harder to show significantly improved prediction [27]. The differences between studies could also be explained by changes in the distribution of risk factors for CVD and their treatment over time, which could differ within and between populations. This has implications for the generalisability of a questionnaire, even within the population in which it was developed, since by necessity the development stage of a risk score has to take place in a different temporal period to the practical application of the score in a clinical setting. It is conceivable that temporal trends in risk factors and the way they are treated could have major impacts on the predictive accuracy of risk scores that were developed using historical data but are applied to predict what may happen in the future. Differences between populations in the underlying distributions of risk factors and their treatment would limit the potential generalisability of risk scores to populations other than that in which they were derived.

It is also important to recognise that the choice of validation population will have an influence on the performance of a risk score in estimating CVD risk. Studies that develop risk scores in one half of a cohort and then validate them in the other half are likely to report better predictive abilities. This is true of scores developed from the Hong Kong Diabetes Registry [18, 26] and the Swedish National Diabetes Register [23]. Conversely, validating risk scores in different populations and ethnic groups is likely to result in relatively poorer prediction. For example, the UKPDS risk engine had moderate discrimination and poor calibration when evaluated in a Chinese diabetic population [18]. This underlines the fact that the accuracy of a risk score largely relies on the background risk of a specific population to which it is applied. It may be more useful to develop or recalibrate population-specific risk prediction tools, rather than trying to find a universal risk score that will work in all populations.

This review used a comprehensive search without language restrictions. To the best of our knowledge, it is the first systematic review to assess the ability of risk scores to estimate cardiovascular risk in individuals with diabetes. We extracted data on widely used measures of predictive accuracy, including discrimination, calibration, sensitivity, specificity and positive and negative predictive values. Apart from a difference in the choice of validation population, each study had different inclusion criteria, follow-up, ascertainment methods, definition of diabetes and CVD endpoints. This makes it difficult to compare the predictive ability between different risk scores and between validation populations. The performance of CVD risk scores varied considerably, and there is no conclusive evidence of a difference in the predictive performance of risk scores developed in individuals with diabetes and of those developed in the general population. A variety of statistical approaches was used to describe and compare the predictive performance of the different risk scores; our review suggests that a more systematic and standardised approach is needed.

Methodological issues

The estimation of CVD risk is a dynamic research field and there are a number of unresolved methodological issues concerning the development, validation and comparison of risk scores. It is clear from the systematic review that a number of statistical techniques can be employed to assess the performance of predictive models. However, these measures are not uniformly calculated across all studies and it is sometimes difficult to obtain a comprehensive picture of the use of a risk score without taking all these measures into account. Risk scores are frequently derived using a logistic regression model, which models the log odds of having the event during the specified follow-up period as a function of the other variables. The estimated coefficients from the logistic regression model are then used to define the risk score. Although this risk estimate is continuous, many of the methods of comparing prediction assume that a threshold will be used to categorise people into two groups on the basis of the risk score.

Measures of discrimination Discrimination is the ability of the prediction model to correctly separate individuals into those who will and those who will not have the event of interest. The commonly used receiver operating characteristic curve (ROC) is a plot of sensitivity on the vertical axis against (1−specificity) on the horizontal axis for every possible cut-off value of the continuous risk score. The aROC is the area under the ROC curve and is equal to the probability that a randomly selected individual with the event has a higher value of the risk score than a randomly selected individual without the event. A test that was totally uninformative would have a c value of 0.5, while a perfect test would have a c value of 1. The discrimination of two possible models can be compared by comparing the values of their aROCs. Net reclassification improvement (NRI) [28] is used to compare two models A and B, which share all risk factors except for one new marker included in model B. After classification of the predicted probabilities from the two models into risk categories, the NRI is the proportion of times model B correctly moves an individual with the event into a higher risk category or an individual without the event into a lower risk category, minus the proportion of times the new model incorrectly moves an individual with the event into a lower risk category or an individual without the event into a higher risk category. Integrated discrimination improvement (IDI) [28] is calculated by subtracting, for model A and B, the average probability of the event in the individuals without the event from the average probability of the event in the individuals with the event, and then calculating the difference in these two quantities. The values of NRI and IDI are both between 0 and 1, or 0 and 100%. The higher the values, the greater the improvement in performance of the new model. It is possible to test whether each statistic is significantly different from 0.

Calibration of risk scores Calibration is the extent to which predicted risk from a model equals observed risk in the data. The Hosmer–Lemeshow Goodness of fit test is calculated by ordering the predicted probabilities of the event into, say, ten near equal sized groups. The observed and expected numbers of individuals with the event within each group are then compared using a χ2 statistic (with nine degrees of freedom if there are ten groups). A statistically significant result implies that the model may be poorly calibrated.

Global measures of model fit The likelihood ratio statistic is a global measure of model fit. The Bayesian information criterion (BIC) and Akaike’s information criterion (AIC) are two measures that combine both fit (as measured by the likelihood ratio statistic) and model complexity, in terms of number of variables (BIC and AIC) and also (for BIC) number of observations. These measures can be used in the same dataset for model selection; comparing two models, the model with the lower value of either BIC or AIC would be preferred. However, as Ware [29] has demonstrated, a new marker could be an important risk factor that significantly improves the fit of the model, but it may have almost no impact on model discrimination as measured by aROC.CVD risk scores are often compared using the aROC, with a significant increase in the aROC being taken as evidence that the discrimination of a risk score has improved. More recently, Cook and others [30, 31] have argued that the aROC is an insensitive measure based purely on ranks and that we should also calculate a measure of calibration (e.g. the Hosmer–Lemeshow statistic) and global fit (e.g. BIC) in order to comprehensively assess the utility of a risk score. They argue that the critical issue for clinical application is ‘the proportion of patients reclassified using a new risk algorithm and whether the magnitude of this reclassification is large enough to alter physician behaviour with regard to prevention’ [31, 32]. Pepe et al. [33] recently suggested calculating the NRI and IDI measures when comparing two scores, in order to assess whether any reclassification was in the right or wrong direction.An example from our previous research illustrates some of these analytical approaches [34]. Using data from EPIC-Norfolk, a UK population-based prospective cohort [35], we computed two novel risk scores by fitting Cox proportional hazards regression models with CHD as the outcome. In model A, we used the original Framingham risk score variables, i.e. age, total cholesterol, HDL-cholesterol, systolic blood pressure, smoking status and diabetes. In model B, we replaced the diabetes variable with HbA1c to examine whether the addition of this continuous variable improved the prediction of CHD. While the results on discrimination indicated that model B was better than model A at distinguishing between individuals with and without the disease, results from the NRI statistic suggested that this difference did not significantly improve reclassification. This conflicting information highlights the challenges of comparing CVD risk scores and demonstrates that despite statistical advances, a number of questions remain unanswered.Most of the risk scores in this systematic review were developed using models for censored survival time data, such as the Cox proportional hazards model or the accelerated failure time model. However, the most common method for deriving the aROC is based on a logistic regression model, which ignores the time-dependent nature of the data. New approaches to assessing the performance of a risk score estimated from survival data in the presence of censoring have been proposed [36, 37]. These new performance measures have been shown to be unbiased, unlike the aROC from logistic regression, which tends to be underestimated and have a larger standard deviation [37].

Conclusions

The computation of CVD risk is likely to remain an important part of the process of prioritising therapy for individuals and populations. The degree to which such risk scores can be improved is questionable, as attempts to add novel risk factors to existing CVD scores and thereby improve their predictive ability have not been very successful [38, 39], largely because a risk factor must be very strongly associated with a disorder for it to be useful for prediction [40]. Few novel biomarkers have demonstrated an ability to predict risk over and above information available from global assessment tools such as Framingham [38, 41]. Furthermore, there is rarely evidence that reductions in any of these novel markers will lower cardiovascular risk [42]. Genetic risk information currently adds little to prediction, but may become increasingly important in the future. However, there is little evidence that the provision of genetic risk information is associated with behaviour change [43].

Thus while there is clearly some scope for improving prediction, it is likely that improvements will be marginal. Indeed, it may be more profitable to focus on ensuring that tools currently available for risk prediction are applied more broadly and routinely throughout clinical practice in order to address the gap between the promise of CVD prevention and its reality [39]. When attempting to reduce CVD risk, the precision of the instrument may be less important than how it is used. As such, there is still a need for further research into provider and patient perceptions of CVD risk. There is considerable uncertainty on how best to present risk information and on whether the presentation of risk information is associated with lifestyle change or the degree of medication adherence [44-46]. The downsides of presenting risk information also need to be considered. For example, it is possible that showing how little CVD risk is reduced by lowering blood glucose alone may discourage people from adhering to hypoglycaemic medication.

Cardiovascular risk scores are useful tools in the management of individuals with diabetes, particularly when the score has been developed in a population of similar individuals. Scores that rank risk well are appropriate for identifying those at highest risk, to whom therapy can then be targeted. Conversely, the process of predicting risk accurately in order to provide prognostic information is better aided by risk scores that accurately quantify absolute risk. Finally, we see a potentially important role for scores capable of quantifying that element of risk that is modifiable, a strategy that could help motivate patients to change. While improvement in the predictive ability of risk scores might still be obtained, the public health utility of their application depends on a far wider range of issues.

#1 risk assessment [Mesh]	#20 coronary disease risk assessment	#39 Reynolds risk
#2 predictive value of tests [Mesh]	#21 chd risk assessment	#40 Dundee risk
#3 Sensitivity and Specificity [Mesh]	#22 cvd risk assessment	#41 brhs score
#4 Reproducibility of Results [Mesh]	#23 heart disease risk assessment	#42 British Family Heart Study
#5 risk score	#24 cvd risk score	#43 New Zealand chart
#6 risk calculat*	#25 OR/15–24	#44 seven countries study
#7 risk engine	#26 framingham risk	#45 Progetto CUORE
#8 risk equation	#27 framingham risk*	#46 OR/26–45
#9 risk algorithm	#28 framingham equation	#47 14 OR 25 OR 46
#10 risk chart	#29 framingham model	#48 cardiovascular disease [Mesh]a
#11 risk table	#30 PROCAM	#49 coronary artery disease [Mesh]b
#12 risk prediction	#31 score	#50 stroke [Mesh]c
#13 risk functions	#32 decode	#51 OR/48–50
#14 OR/1–13	#33 ukpds	#52 diabetes mellitus, type 2 [Mesh]d
#15 cardiovascular risk score	#34 copenhagen risk	#53 51 AND 52
#16 cardiovascular risk assess*	#35 British Regional Heart Study	#54 Diabetic Angiopathies/epidemiology [Mesh]
#17 cardiovascular risk assessment	#36 Swedish National Diabetes Register	#55 Diabetic Angiopathies/etiology [Mesh]
#18 cardiovascular disease risk assessment	#37 Tayside	#56 53 OR 54 OR 55
#19 coronary risk assessment	#38 qrisk	#57 47 AND 56