External Validation of the Prognostic Prediction Model for 4-Year Risk of Metabolic Syndrome in Adults: A Retrospective Cohort Study.

PURPOSE: A prediction model for 4-year risk of metabolic syndrome in adults was previously developed and internally validated. However, external validity or generalizability for this model was not assessed so it is not appropriate for clinical application. We aimed to externally validate this model based on a retrospective cohort. PATIENTS AND METHODS: A retrospective cohort design and a temporal validation strategy were used in this study based on a dataset from 1 January 2015 to 31 December 2018. Multiple imputation was used for missing values. Model performance was evaluated by using discrimination, calibration (calibration plot, calibration slope, and calibration intercept), overall performance (Brier score), and decision curve analysis.
RESULTS: In external validation, the C-statistic was 0.782 (95% CI, 0.771-0.793). The calibration plot shows good calibration, calibration slope was 1.006 (95% CI, -0.011-1.063), and calibration intercept was -0.045 (95% CI, -0.113-0.022). Brier score was 0.164.The discrimination and calibration of the prediction model were good in temporal external validation.
CONCLUSION: The discrimination and calibration of the prediction model were satisfactory in the temporal external validation. However, clinicians should be aware that this prediction model was developed and validated in a tertiary setting. It is strongly recommended that further studies validate this model in international cohorts and large, prospective cohorts in different institutions.

Introduction

Metabolic syndrome (MetS) is defined as a cluster of cardiometabolic risk factors including abdominal obesity, dyslipidemia, hypertension, and insulin resistance.1 The literature revealed that the global prevalence of MetS was around 25% among adults in 2015.2 MetS is becoming an important health concern, since it results in alarming diabetes mellitus prevalence rates and the increased risk of cardiovascular disease (CVD).3 Consequently, metabolic syndrome has caused a heavy economic burden for healthcare and social systems. It is imperative to promote early detection and provide interventions to improve patients’ future cardiometabolic risk and outcomes.

In public health and clinical practice domains, prognostic prediction models can screen individuals that are at a relatively high risk of developing certain diseases in the future and help physicians make therapeutic decisions (eg, lifestyle changes) based on the probability of a prognostic outcome.4 Thus, it is necessary to use the prognostic prediction model for adults at risk of metabolic syndrome.

We previously conducted a systematic review by searching four English databases and four Chinese databases, and used the prediction model risk of bias assessment tool (PROBAST) to perform the critical appraisal for the prognostic prediction models of metabolic syndrome.5 The results suggested that existing prognostic prediction models for metabolic syndrome were not yet suitable for clinical application due to the high risk of bias in the methodological quality of the models.

Thus, we developed and internally validated a prediction model for 4-year risk of metabolic syndrome in adults based on a previous health examination cohort.6 Although this model has been internally validated and reproducibility of this predictive model has been determined, external validity or generalizability was not assessed, deeming it as inappropriate for clinical application. External validity ensures that the model is validated in samples that have not been used in the model development process and internal validation.7

To prompt the application of the prognostic prediction model for 4-year risk of metabolic syndrome in adults in practice, we aimed to externally validate this model based on a retrospective cohort in China.

Patients and Methods

Study Design and Data Source

A retrospective cohort design and a temporal validation strategy were used in this study. Temporal validation means that data are collected from a later time period in the same setting. The prediction model was externally validated using the same predictors, outcome definitions, and measurements. We developed and internal validated the prediction model for 4-year risk of metabolic syndrome in adults based on dataset from 1 January 2011 to 31 December 2014. For external validation, a retrospective cohort of health examination from January 2015 to 31 December 2018 from the healthcare information and management systems of a tertiary hospital was obtained. For this study, these inclusion criteria were used: (1) participants of at least 18 years of age or older, (2) participants without metabolic syndrome in 2015, (3) participants attend health examination for 4 years consecutively.

Outcomes

The outcome was defined as metabolic syndrome (MetS), and the diagnostic criteria was 2009 Joint Scientific Statement (harmonizing criteria 2009).8 This criteria includes five risk factors: ① waist circumference(WC)≥85 cm (male); waist circumference≥80 cm (female) ② Triglycerides (TG)≥1.7 mmol/L or treatment; ③ Plasma high-density lipoprotein cholesterol (HDL-C) <1.0 mmol/L (male) or treatment; Plasma high-density lipoprotein cholesterol <1.3 mmol/L (female) or treatment; ④ Systolic blood pressure(SBP)≥130 and/or diastolic blood pressure(DBP)≥85 mmHg or treatment ⑤ Fasting plasma glucose (FPG)≥5.6 mmol/L or treatment. According to harmonizing criteria 2009, when any three of five factors were present, participants could be diagnosed with MetS.

Predictor Variables

The prediction model for 4-year risk of metabolic syndrome in adults included five independent predictors. These predictors were age(years), total cholesterol (mmol/l), serum uric acid (μmol/l), alanine transaminase (U/L), and body mass index (Kg/m2). Related information is available in the dataset and can be extracted from the dataset.

Missing Data

Multiple imputation was used for missing values by using chained equations on the basis of a missing at random assumption. The number of imputed datasets was five. When conducting multiple imputation, the predictor variables and the outcome were included in the imputation model.

Sample Size

It is recommended that a minimum of 100 events for external validation samples using logistic regression models is needed.9 This is because a validation study aims to obtain an accurate and precise estimation of model performance and avoid the risk for biased estimates of model performance in a different dataset.10

Model Performance

For external validation, we adopted the same metrics of model performance used in previous internal validation. They were discrimination, calibration, overall performance, and decision curve analysis. For discrimination, the c-statistic was used.11 Normally, C-values of 0.7 to 0.8 indicate acceptable discrimination, C-values of 0.8 to 0.9 indicate excellent discrimination, and C-values of ≥0.9 indicate outstanding discrimination.12 For calibration, the calibration slope and intercept were calculated.13 Additionally, the calibration plot using a locally weighted scatterplot smoothing was created to visualize the agreement between the mean predicted risk and the observed events.14,15 It is suggested that ideal values of calibration intercept and calibration slope should be 0 and 1, respectively.12 Brier score was chosen for overall performance, as it can capture both discrimination and calibration.16 The Brier score of flipping a fair coin is 0.25.17 Decision curve analysis was then used to estimate the net benefit of using the prediction model for 4-year risk of metabolic syndrome in adults. This is because true positives and false positives generated by the model can be compared using decision curve analysis, and subsequently, the model’s expected utility in clinical practice can be estimated.

Statistical Analysis

We adopted the same approach used in the internal validation study to deal with predictors in this study. In the model development and internal validation stage, we adopted the least absolute shrinkage and selection operator (LASSO) regression analysis to avoid overfitting and multicollinearity for predictor selection, so all predictors in the previous model were kept for external validation. Continuous variables were not dichotomized. We kept them as continuous variables in the model to avoid losing prognostic information. If the relationship of continuous predictors with the outcome was nonlinear, fractional polynomials (FP) was used. Lastly, a logistic model was fitted with the predictors and outcome.

Ethical Approval

This study was approved by the ethics committee of Zhejiang University School of Medicine Sir Run Run Shaw Hospital (20181220–3). The ethics committee approved the request to waive the documentation of informed consent due to secondary use of existing data. This study was conducted in accordance with the principles of the Declaration of Helsinki. The datasets presented in this article are not readily available, because the datasets were protected and confidential according to the hospital policy.

Results

Baseline Characteristics

A total of 7681 participants were included according to the inclusion and exclusion criteria (Figure 1). After 4 years of follow-up, 2222 participants were diagnosed with MetS. EPV was above 400, so the sample size in this study was reasonable. The prevalence of metabolic syndrome in adults was 28.93%. Table 1 shows the characteristics of the participants.

Figure 1

Selection of participants.

Table 1

Characteristics of Participants

Candidate Predictor	Validation Cohort (N=7681) n (%)/Mean±SD	Missing Values n (%)
Sex
Female	3230(42.05%)	0
Male	4451(57.95%)	0
Age	41.988±11.246	0
WBC	5.8±1.431	0.078
Hb	140.456±14.781	0.078
LC	33.498±7.298	0.091
NGC	3.325±1.122	0.104
TC	4.834±0.893	0.234
UA	331.758±83.226	0.234
Weight	62.816±10.435	31.194
Height	165.988±7.897	31.181
BMI	22.697±2.681	31.194
MCV	90.207±5.056	0.078
MCH	30.459±1.988	0.078
HCT	41.602±4.213	0.078
AST	20.993±12.366	40.737
ALT	20.984±15.619	0.169
Outcome indicator	Participants with events (N=2222) n (%)/Mean±SD	Missing Values n (%)
WC	86.445±7.327	17.057
TG	1.83±1.165	0.045
HDL-c	1.183±0.3	0.045
SBP	125.101±14.657	3.375
DBP	75.841±10.361	3.375
FPG	5.324±0.762	0.045

External Validation

In external validation, the C-statistic was 0.782 (95% CI, 0.771–0.793). The calibration plot shows good calibration (Figure 2), calibration slope was 1.006 (95% CI, −0.011–1.063), and calibration intercept was −0.045 (95% CI, −0.113–0.022). Brier score was 0.164. Figure 3 indicates the decision curves and net benefit of using the prognostic prediction model for identifying patients with MetS. It is suggested that compared to treating no patients or all patients, greater net benefit was provided by using this prediction model. Additionally, compared to using a single predictor in clinical practice, the results of the decision curve analysis suggested that the prognostic prediction model could obtain maximum net benefit.

Figure 2

Assessing calibration in the external validation cohort.

Figure 3

Decision-curve analysis with net benefit by threshold probability.

Clinical Examples

A web-based calculator was developed to predict individuals with risk of developing MetS in 4 years, and the webpage was . This web-based calculator could facilitate the clinical application, and we also provided a clinical example of the application of the web-based calculator (Figure 4).

Figure 4

A clinical example for using the web-based calculator to predict the risk of developing MetS in 4 years.

Discussion

Decision guidance for adopting intervention and prevention for individuals at risk of developing MetS in the future by using an externally validated prediction model can optimize treatment processes and decrease the use of medical resources. Following the TRIPOD guidelines, this prediction model was internally and externally validated on two different datasets. In this external validation study, we found that this prediction model retained good discrimination and calibration. Moreover, Brier score indicated that the overall performance was acceptable As is often the case, prediction tools perform worse in external validation than in the original derivation dataset. This is because, for internal validation, the prediction models are validated in the dataset that is used for developing the models.

In the internal validation, the prevalence of metabolic syndrome in adults was 25.76%. In the external validation, the prevalence of metabolic syndrome was 28.93%, which was higher than the number in internal validation. According to previous research, the prevalence of MetS was around 25% among adults and our results were similar.2 The C statistic in external validation (0.782) did not drop steeply than in internal validation (0.783). Comparisons between the original derivation dataset and external validation indicated that patient’s characteristics were broadly similar. For example, in external validation, the proportion of males was 57.95%, whereas, for development and internal validation, it was 65.01%.6 Additionally, the mean age was 41.988 in external validation, whereas for development and internal validation, it was 41.488.6

According to our previous systematic review, 11 prognostic prediction models were developed for individuals at risk of developing MetS.5 However, only 3 studies conducted external validation. Yang et al found that the C statistic were above 0.8 in external validation, and this value was larger than the value in our study (0.782).18 Compared to our model, the predictors in their model were included in the outcome definition. This could cause the relationship between predictors and outcomes to be overestimated, and result in an optimistic performance of the model. Pujos-Guillot et al did not report discrimination and calibration in external validation.19 Efstathiou et al reported sensitivity, specificity, and the results of Hosmer–Lemeshow test.20 However, the C statistic are more appropriate than sensitivity and specificity when reporting discrimination for prediction models, because without a predefined probability threshold, the model performance could be optimistic.21 Additionally, according to TRIPOD, there are some limitations in Hosmer–Lemeshow test. First, Hosmer–Lemeshow test cannot provide information on the magnitude of the difference or the agreement of values predicted by the model among individuals with low and high risk. Second, if the sample size is large, a clinically trivial difference between the predicted and the observed risk could lead to a statistically significant result in Hosmer-Lemeshow test. Thus, Hosmer-Lemeshow test could mislead the results of model calibration.

In our study, we adopted the C statistic as the index of discrimination. Moreover, calibration is a multidimensional concept, so a single statistic cannot be sufficient to summarize calibration. This study used calibration plot, calibration slope (the overall prognostic effects of predictors in the model), and calibration intercept (the expected proportion of MetS when all the predictors in the model are zero) for model calibration evaluation. Additionally, we adopted decision curve analysis to estimate the net benefit of the model in clinical practice. When using the prognostic prediction model, the results of the decision curve analysis in the external validation cohort indicated that greater net benefit was provided than the default strategies of treatment for all individuals or for no individuals.

In the context of prognostication, predictions can be used to make decisions on the basis of the risk for developing a particular outcome within a specific period of time in clinical practice.22 The early prediction of developing MetS in adults by using the prognostic prediction model could facilitate the use of non-drug preventive measures in high-risk individuals. Lifestyle intervention can be viewed as the most important non-drug preventive measure for MetS. According to a systematic review, receiving dietary intervention and supervised exercise is effective in reducing the individual risk factors for metabolic syndrome and reducing the prevalence of metabolic syndrome.23 The use of this prediction model to identify high-risk individuals could offer a vital contribution to clinical practice and ensure the efficient use of medical resources for high-risk individuals. Additionally, the modifiable risk factors of the model may facilitate the use of preventive measures. Future research should focus on studying the optimal cut-off point of this prediction model.

Limitations

This study presents some limitations worth considering. Due to the retrospective study design, the dataset suffers from missing data and different collections and records from the derivation cohort.24 Studies that collect data prospectively to avoid missing data are needed in the future. Additionally, this prediction model was developed and validated externally in a single institution in China. Geographic validation and international validation in multiple countries are recommended. Lastly, clinicians should be aware that this prediction model was developed and validated in a tertiary setting. Thus, applicability to patients in primary settings remains to be determined. It is suggested that future studies investigate the performance of this model for individuals in primary settings.

Conclusions

This temporal external validation study used a retrospective cohort, and the discrimination and calibration of the prediction model were satisfactory. This means that the model performance in the external validation is satisfied. It is strongly recommended that further studies validate this model in international cohorts and large, prospective cohorts in different institutions. Additionally, future studies could extend the current prognostic prediction model with new predictors which can bring strong incremental values, such as biomarkers and genomics. Ongoing and continuous updating and validating of this prediction model can help improve prognostication in MetS.