Metabolic syndrome does not improve the prediction of 5-year cardiovascular disease and total mortality over standard risk markers. Prospective population based study.

Metabolic syndrome (MS) is widely believed to be an important risk factor for cardiovascular disease (CVD). We assessed whether a model based on MS improved prediction of CVD and total mortality over the Framingham's general CVD system (FRS) and whether MS was better than its individual components. Prospective cohort study of 855 participants randomly selected from the general population. Cox proportional hazards models were used to estimate the hazard ratios selecting a composite endpoint of CVD and total mortality. The performance of the FRS was compared with that of 4 MS-based models that differed in their use of individual components of MS as well as in the use of optimized cut-points of MS. The assessment included metrics of discrimination, calibration, and risk reclassification. Of all the models, only the model containing the 5 optimized components of MS improved model fit (deviance 10.7, P = 0.005), discrimination (difference of areas under the receiving operating curves 0.018), and risk reclassification in participants without events (net reclassification index 5.97, P = 0.01). The addition of optimized waist circumference to the FRS model improved the performance more than any other MS-based model. Every model containing the dichotomous definition of MS failed to improve model fit, discrimination, and risk reclassification. MS did not contribute predictive information over the FRS for the 5-year risk of CVD and total mortality. Some individual components of MS, in particular waist circumference, might play a role as part of the FRS provided their cut-off points are optimized.

INTRODUCTION

There is strong evidence from long-term prospective studies for an association of metabolic syndrome (MS) with the incidence of cardiovascular disease (CVD) and total mortality,[1-4] which has been detected as early as 5 years after diagnosing MS.[5-7] This evidence has led to the often-cited claim that MS is a key risk factor in the prevention of CVD, which has been supported by important scientific societies.[8,9] This claim, notwithstanding the presumed advantage of including knowledge of MS in CVD risk assessment has, to our knowledge, not been demonstrated.[10] In fact, no precise role has been assigned to MS in CVD risk prediction to date.

To incorporate new risk factors into clinical decision making, it is required that these risk factors add predictive information over that offered by established standard methods, such as the Framingham risk score (FRS).[11] This process requires a thorough assessment of not only the strength of the association between the new risk factor and CVD but also of its ability to improve metrics of model performance and risk reclassification of the reference method.[11] In the case of MS, these are issues that have not been defined thus far.[12] Specifically, it has not been addressed what would be the advantage of accounting for MS in a model of risk prediction as compared to a standard reference model, as the FRS. It is also not known whether optimizing the cut-off points of the components of MS would improve prediction over current definitions of MS.[13]

The aim of this study was to investigate to what extent MS added predictive information about the 5-year incidence of new cases of CVD and total mortality over a base model derived from the Framingham's general CVD risk score,[14] as assessed by the standards for the critical appraisal of risk prediction models proposed by the American Heart Association.[11] We further evaluated whether optimizing the cut-off points of the NCEP-ATPIII criteria in our population improved predictions.

METHODS

Study Design and Sampling

This is an observational cohort study consisting of a random sample of participants selected through the local census of the city of Sanlúcar de Barrameda in 2006.[15] Eligible participants were those who were 50 to 75 years old but did not have a malignant neoplasm, an estimated glomerular filtration rate <30 mL/min/1.73 m2, a Child stage B or C cirrhosis, a connective tissue disease, a neurodegenerative disease or a known infection with HIV. Only those participants who had not experienced CVD events were included. To calculate the sample size, we took into account the incidence rate described in Spain of myocardial infarction (135–210 cases/100,000 person-years in men and 29–61/100,000 person-years in women 25–74 years old) and stroke (183–364 cases/100,000 person-years in men and 169/100,000 person-years in women). We also used an estimation of the age and sex-adjusted relative risk of coronary heart disease and stroke attributable to the MS (2.96 and 2.27, respectively).[16,17] The final study sample consisted of a simple random sample of 858 participants from the community. All selected participants were contacted by telephone to be invited to participate in the study and then were seen at the outpatient clinic of our hospital. All participants were enrolled after obtaining informed consent. The protocol for the Sanlúcar Study was evaluated and approved by the Research Ethics Committee of the reference hospital.

Measurement of Risk Factors for CVD at Baseline

All participants were interviewed and underwent physical examination and measurement of height, weight, and waist circumference. Blood pressure was recorded with an OMRON 705CP calibrated monitor with appropriate arm cuff sizes after the participant had been seated for 5 min. The systolic (SBP) and diastolic blood pressure (DBP) for each patient were determined by obtaining the mean of 3 consecutive readings that differed less than 10 mmHg in SBP. After overnight fast, venous blood sample were taken to determine serum levels of total and HDL cholesterol, glucose, and triglycerides using standardized enzymatic methods. Smokers were defined as those who were presently smoking at the time of inclusion in the study or had smoked within the prior 12 months before the study. Diabetes was defined as a fasting glucose of ≥ 126 mg/dL or the current use of hypoglycemic medications, and hypertension was defined as a blood pressure of ≥ 140/90 mmHg or the current use of antihypertensive medications. MS was diagnosed using the modified definition of the NCEP-ATP-III: waist circumference (wc) >102 cm in men or >88 cm in women, SBP ≥ 130 mmHg or DBP ≥ 85 mmHg or taking antihypertensive medication, serum glucose ≥ 100 mg/dL or taking antidiabetic medication, HDL cholesterol <40 mg/dL in men and <50 mg/dL in women, and serum triglycerides ≥ 150 mg/dL.[18]

Follow-Up and Outcomes

We contacted all patients, or their family members in the event of death, after 5 years of follow-up. A total of 685 participants were interviewed at the hospital, 128 were interviewed by telephone, 39 had died, and 3 could not be contacted after the second year of follow-up. The outcome event was defined as a composite endpoint of a first episode of myocardial infarction (with or without ST-segment elevation),[19] stroke (ischemic or hemorrhagic),[20] heart failure[21] or death from CVD and non-CVD causes. Information about the occurrence of events was obtained from clinical reports provided by all participants who had reported events and by reviewing all participants’ medical histories at our hospital. All suspected events were reviewed and confirmed by 3 investigators who were blinded about the patients’ MS status.

Statistical Analysis

Regression Models

Cox proportional hazards models were used to estimate the regression coefficients and hazard ratio (HR) selecting the composite endpoint of CVD and total mortality as the dependent variable. We compared the performance of a base model with that of 4 different enhanced models based on the MS. The base model was derived from the FRS,[14] and included the following covariates: age, sex, smoking, total cholesterol (Ln-transformed), HDL cholesterol, diabetes, and hypertension. In order to avoid multicollinearity, those variables included in the base model that already exist in the definition of MS (diabetes, hypertension, and HDLc) were removed from the MS-based models.[12] Model 1 included age, sex, smoking, total cholesterol, and MS (NCEP-ATPIII); model 2 was the same as model 1 but using its 5 individual components; model 3 was the same as model 1 but using a redefinition of MS with optimized cut-off points for each individual component (see below); and model 4 was the same as model 3 but using the 5 optimized individual components (see below).

The goal of our regression modeling was not to obtain a base model with the best fit but to set a pre-specified model based upon the FRS, which we intended to investigate. Therefore, we retained all pre-specified predictors regardless of their significance level. No significant interaction between covariates was detected.

Metabolic Syndrome With Optimized Cut-Off Points

We modified the cut-off point for each individual component of MS in models 3 and 4 to obtain the optimum performance in sensitivity and specificity as assessed by the Youden index (sensitivity + specificity − 1).[22] The optimized cut-off points in our study population were: wc >112 cm in women and 102 cm in men; SBP >140 mmHg; glucose >122 mg/dL; HDLc <57 mg/dL in women and <50 mg/dL in men; and triglycerides >158 mg/dL. DBP was not included because its contribution in predicting the outcome was not statistically significant.

Assessment of the Fit and Performance of Each Model

Global model fit was assessed by the maximum likelihood method. It was compared across different models by calculating the deviance, defined as the −2 Log-likelihood ratio of each of the investigated models minus -2 Log-likelihood ratio of the base model. The significance level of the deviance was estimated using a Chi-squared test, with degrees of freedom being the difference in the number of parameters between the 2 models. Models with positive and statistically significant deviance improve global model fit compared with the base model.[23]

Model performance was assessed with metrics of discrimination, calibration and reclassification.[11,24,25] Discrimination represents the ability to assign higher predicted risk to participants with events than to participants who did not develop events. Discrimination was compared across models computing the area under the receiver operating characteristic curve (AUC).

Calibration assesses the ability of models to match accurately the observed and predicted level of risk. The Hosmer–Lemeshow X2 test informs how closely the predicted risks agree with the observed risks in each decile of predicted risk. However, in clinical practice, absolute risk is not managed in deciles but in predefined categories of absolute risk. Thus, we also compared calibration of models by stratifying clinically more meaningful categories of absolute risk: <5%, 5–10%, 10–15%, and >15%. The corresponding Hosmer–Lemeshow X2 test with 2 degrees of freedom was calculated. A non-significant P-value is required to accept adequate calibration.

Risk reclassification assesses the proportion of individuals whose predicted risk is correctly reclassified into different risk categories using the MS-based models compared with the base model. Ideally, a good model should reclassify the cases’ predicted risk upward and the non-cases’ predicted risk downward.[26] We calculated the net reclassification improvement (NRI) by cross-tabulating the predicted risk of the MS-based models against the predicted risk of the base model categorized in 4 categories of risk: <5%, 5–10%, 10–15%, and >15%. The NRI was displayed separately for cases and non-cases. Positive values of NRI would indicate adequate reclassification of risk, whereas negative values indicate inadequate reclassification of risk. Finally, we included an estimation of the integrated discrimination improvement (IDI) that does not require a definition of categories of risk. It was calculated as the difference in the discrimination slope of MS models and the base model. The discrimination slope of each model was the difference in the average predicted risk for cases minus the average predicted risk for non-cases. The higher the IDI, the better the discrimination between cases and non-cases by the new model.

Two-sided P values of less than 0.05 were required for statistical significance. Analyses were conducted using SPSS 15.0 (Chicago, IL).

RESULTS

The characteristics of the study participants at baseline are described in Table 1. We obtained complete information on the occurrence of events in 855 (99.6%) participants after a median follow-up period of 60.5 months. The remaining 3 participants were not included in the analysis. A total number of 88 cases had the outcome (16 cases had a myocardial infarction without ST-elevation, 5 cases had a myocardial infarction with ST-elevation, 7 cases had an ischemic stroke, 2 cases had a hemorrhagic stroke, 19 cases had acute heart failure, and 39 cases died; 10 from cardiovascular causes and 29 from non-cardiovascular causes). The adjusted regression coefficients and HR for the incidence of CVD and death in the base model, and the MS-based models are shown in Table 2.

TABLE 1

Baseline Characteristics of the Study Sample According the Occurrence of Events at 5 Years

TABLE 2

Coefficients and Hazard Ratios of the Reference Model and MS-Based Models for 5-Year CVD and All-Cause Mortality

Every model showed adequate calibration both in deciles of predicted risk and more importantly, in the 4 strata of absolute risk (Table 3). However, MS in model 1 or its individual components in model 2 worsened the model fit and the discrimination of predicted risk between cases and non-cases compared with the base model (Table 3). This occurred despite the fact that MS was independently associated with the outcome beyond its 5 individual components in model 2 (Table 2). Compared with the base model, the individual components of MS in model 2 produced an incorrect net reclassification of predicted risk for both cases and non-cases. This model reclassified 8% of cases incorrectly into lower categories of risk (P = 0.24) and 6% of non-cases into higher categories of risk (P = 0.009, Table 4). Very similar results were obtained using the NCEP-ATPIII definition of MS. When discrimination was assessed by the IDI, both models 1 and 2 also produced an inadequate and statistically significant reduction in the difference of the 5-year mean predicted risk between cases and non-cases compared with the base model (Table 4).

TABLE 3

Performance Summary of the Base Model and MS Models for 5-Year CVD and All-Cause Mortality

TABLE 4

Risk Reclassification Summary for Cases and Non-Cases by MS Models Over the Base Model

The most remarkable findings were obtained after optimizing the cut-off point of MS. Glucose, wc and HDLc moved from a non-significant association with the outcome in model 2 to a highly significant association in model 4 (P = 0.01, 0.02, and 0.004, respectively, Table 2). The optimized components of MS in model 4 did improve global fit (deviance 10.7, P = 0.005, Table 3) and discrimination of risk between cases and non-cases as assessed by the AUC, although the difference of areas was not statistically significant (P = 0.63, Table 3, Figure 1). More importantly, only the optimized components of MS achieved a correct net reclassification of risk that was not significant for cases but statistically significant for non-cases (net reclassification of non-cases 6%, P = 0.01, Table 4). As compared with the base model, the optimized definition of MS in model 3 failed to improve model fit, discrimination, calibration and risk reclassification, and did not perform better than its individual components in model 4 (Tables 3 and 4). Even more, the single inclusion in the base model of the optimized wc provided a better global fit, model performance and risk reclassification than the base model alone and any other MS-based model (deviance 13.186, P = 0.001; AUC 0.775, 95% CI 0.725–0.825; calibration in strata of absolute risk, P = 0.54; net reclassification of cases 2.27%, P = 0.73; net reclassification of non-cases 8.70%, P < 0.001; IDI 2.11%, P < 0.001).

FIGURE 1

Area under the receiver operating characteristic curve of the base model and MS models.

We also reproduced the analysis selecting incident CVD, excluding total mortality, as the dependent variable. Despite the fact that MS increased HR for incident CVD, metrics of global fit and discrimination worsened in all MS-based models. In addition, all models except model 4 (optimized individual components of MS) produced a statistically significant incorrect net reclassification of risk for cases (−28.8%, P = 0.003; −28.8%, P = 0.002; −27.1%, P = 0.008; −3.4%, P = 0.71 for Models 1–4, respectively). Models 3 and 4 achieved a non-significant increase in the net reclassification of non-cases (1.9%, P = 0.39; 0.7%, P = 0.73, respectively). However, MS in Model 1 or its components in Model 2 worsened the net reclassification of non-cases (−7.6%, P = 0.001; −3.9%, P = 0.08). All MS models worsened the IDI (−5.0%, P < 0.001; −4.3%, P < 0.001; −4.0%, P = 0.001; −1.3% P = 0.26 for Models 1–4, respectively).

DISCUSSION

This study did not demonstrate that the NCEP-ATPIII definition of MS improve predictive information about the occurrence of 5-year CVD and total mortality over a base model derived from the FRS. Our results do not support the strategy of including MS in clinical decision making for the purpose of risk stratification. Moreover, MS was worse at discriminating the predicted risk between cases and non-cases as well as in reclassifying people who developed and who did not develop events into more appropriate categories of predicted risk. The most remarkable practical consequence of accounting for MS in risk stratification would be a less intensive intervention in 9% of subjects with events and unnecessary intensive risk treatment in another 7% of truly low risk subjects that were incorrectly reclassified into higher categories of risk.

The association of MS and CVD outcomes has been heavily evaluated in the last decade. But demonstrating association is the most basic requirement but not the last one.[11] The next requirement is to demonstrate that MS is able to improve risk prediction beyond established predictive risk systems through a complete assessment of discrimination, calibration and risk reclassification.[11] Assessment of diverse metrics of model performance is of critical value as no single statistical measure provides all the information needed to analyze the contribution of novel risk factors.[11] To our knowledge, there is no single study that has simultaneously addressed all these statistical measurements. Most previous studies have centered their analysis on isolated methods of model performance, most frequently using metrics of discrimination based on the AUC.[27] However, it is widely recognized that the AUC may underestimate the contribution of individual variables added to standard models to risk prediction. For this purpose, metrics based on risk reclassification are more suitable.[26]

The FRS is a predictive tool designed to identify people at high-risk for any atherosclerotic CVD event.[14] The accuracy of this risk function has been shown to be related to the background risk of the population to which it is applied.[28] Accordingly, the FRS may overestimate the true cardiovascular risk in low-risk populations, such as in populations in many Mediterranean countries,[28] as a consequence of difficulties in extrapolating predictions to different populations and the limited risk factor set included in the score.[29,30] The incorporation of new risk factors into the FRS may improve its model performance.[30]

Improvement in risk reclassification for the incidence of new CVD events has recently been assessed using several risk markers, but not the MS, added to the FRS.[31] Our study is the first in which MS is subjected to a complete appraisal about its performance including risk reclassification. In the San Antonio Heart Study, combining MS with the FRS did not improve the prediction of CVD at 7.5 years as assessed solely by the corresponding models’ sensitivity and specificity.[32] The contribution of MS to risk reclassification has previously been analyzed in a study that focused on patients with overt vascular disease who were followed for a median of 3.7 years. In that study, MS was unable to improve risk reclassification of events over a model that only included age and sex.[33] More recently, a study using factor analysis found a 7% improvement in the IDI for an extended definition of MS that included C-reactive protein over the traditional definition that included all 5 components. No comparison with a base model of standard risk markers was reported.[34]

Interest in MS as a predictor of CVD derives from large studies that demonstrated an increased risk of CVD after accounting for traditional risk factors and also for the FRS.[2,35] Our study illustrated that MS was unable to improve the performance of the FRS despite its independent association with incident CVD and mortality. This can be explained by taking into account that the optimum cut-off point of MS in our population was distant from that of the NCEP-ATPIII criteria.[26] The most striking difference was an increase of 24 cm in the optimized value of wc in women, and as a result of its sole addition to the base model, this model performed better than any other MS-based model.[36] This finding highlights the importance of optimizing the cut-off point values for the risk factors when evaluating precise outcomes, which is of maximum interest when prediction models are applied to populations with different backgrounds of CVD risk.

Our study has several limitations. We cannot rule out that the prediction ability of MS may be different in populations with other CVD risk backgrounds. Even so, it is unlikely that the prediction ability of the dichotomous definition of MS exceeds that of its individual components. Our objective of interest was to predict a composite endpoint of myocardial infarction, stroke, heart failure, and total mortality. It is likely that the prediction ability of MS may differ for each separate endpoint. However, taking into account the commonality of risk factors specific for CVD, it is also likely that the risk prediction of a global CVD endpoint parallels that of its individual components.[14] The results of this study are also applicable to populations of individuals aged 50 to 75 years old and for a time length of risk prediction of 5 years. Smoking and total cholesterol did not validate in our base model. In the case of smoking, we did not account for the lifetime tobacco exposure, which could have affected the association between smoking and the outcome. In the case of total cholesterol, 24.9% of all participants were taken hypolipemic therapy at the time of inclusion. These participants had a much more unfavorable CVD risk profile than those who were not taking hypolipemic medication despite the fact that they presented lower serum lipid levels at baseline. Our results may not be extrapolated to people of other ages and for different periods of follow-up. The FRS has been validated to predict risk for a time period of 5 years, and we were interested in assessing the usefulness of including MS within the same time frame.[37,38] Finally, as results of risk reclassification are affected by the number of strata and the cut-off points for each stratum, we focused on ranges of risk with clinical meanings comprising between <5% and >15% of absolute risk.[39]

Based upon our results, we concluded that the NCEP-ATPIII definition of MS did not improve predictive information over the FRS and that there was no advantage in transforming its 5 individual components into a dichotomic output of yes/no for MS. Future research in populations with different CVD risk profiles might establish roles for some individual components of MS, such as wc, as components of the prediction model, provided their cut-off points in men and women are optimized for the outcome.