A clinical diagnostic model based on an eXtreme Gradient Boosting algorithm to distinguish type 1 diabetes.

BACKGROUND: Accurate classification of type 1 diabetes (T1DM) and type 2 diabetes (T2DM) in the early phase is crucial for individual precision treatment. This study aimed to develop a classification model having fewer and easier to access clinical variables to distinguish T1DM in newly diagnosed diabetes in adults.
METHODS: Clinical and laboratory data were collected from 15,206 adults with newly diagnosed diabetes in this cross-sectional study. This cohort represented 20 provinces and 4 municipalities in China. Types of diabetes were determined based on postprandial C-peptide (PCP) level and glutamic acid decarboxylase autoantibody (GADA) titer. We developed multivariable clinical diagnostic models using the eXtreme Gradient Boosting (XGBoost) algorithm. Classification variables included in the final model were based on their scores of importance. Model performance was evaluated by area under the receiver operating characteristic curve (ROC AUC), sensitivity, and specificity. The performance of models with different variable combinations was compared. Calibration intercept and slope were evaluated for the final model.
RESULTS: Among the newly diagnosed diabetes cohort, 1,465 (9.63%) persons had T1DM and 13,741 (90.37%) had T2DM. Body mass index (BMI) contributed the most to the model, followed by age of onset and hemoglobin A1c (HbA1c). Compared with models with other clinical variable combinations, a final model that integrated age of onset, BMI and HbA1c had relatively higher performance. The ROC AUC, sensitivity, and specificity for this model were 0.83 (95% CI, 0.80 to 0.85), 0.77, and 0.76, respectively. The calibration intercept and slope were 0.02 (95% CI, -0.03 to 0.06) and 0.90 (95% CI, 0.79 to 1.02), respectively, which suggested a good calibration performance.
CONCLUSIONS: Our classification model that integrated age of onset, BMI, and HbA1c could distinguish T1DM from T2DM, which provides a useful tool in assisting physicians in subtyping and precising treatment in diabetes. 2021 Annals of Translational Medicine. All rights reserved.

Introduction

Treatment of diabetes mellitus requires accurate discrimination of type 1 diabetes (T1DM) and type 2 diabetes (T2DM) (1). Diabetes classification systems have evolved in recent years. Updated classification criteria recommended in 2019 by the World Health Organization propose subtypes of hybrid diabetes and unclassified diabetes in addition to the well-known T1DM, T2DM, gestational diabetes, and other types of diabetes (2). Ahlqvist et al. stratified adult-onset diabetes into five subgroups that had different disease progression and risks of complications (3). Among these five clusters, patients in cluster 1 were denoted as having severe autoimmune diabetes (SAID); cluster 2 was labeled as severe insulin-deficient diabetes (SIDD) that required rapid insulin treatment. In addition to different typing strategies, key clues for the discrimination of T1DM from the majority of patients with T2DM are the presence of diabetes-associated autoantibodies and a deficiency of β cell function characterized by low plasma C-peptide level.

C-peptide is a useful method for assessing β cell function. C-peptides more than 0.3 nmol/L could differentiate insulin-requiring from non-insulin-requiring diabetes (4,5). Our group recently showed that glutamic acid decarboxylase autoantibody (GADA) titer is a valid risk predictor for progression of beta-cell failure in adult patients with autoimmune diabetes (6). About 70% of the patients in high GADA titer (more than 173.5 U/mL) group progressed to beta-cell function failure during the follow-up period. Therefore, C-peptides less than 0.3 nmol/L and GADA titer higher than 173.5 U/mL could be reliable evidence for T1DM identification.

The measurement of GADA and C-peptide in every individual with diabetes can be challenging and costly, especially for clinics in developing or undeveloped countries. Thus, efforts have been made to identify T1DM from T2DM from clinical parameters obtained in routine examinations. For instance, the UK Practical Classification Guidelines for Diabetes proposed that age of diagnosis (35 years as the cut off) and insulin treatment could be used to discriminate T1DM and T2DM (7). Fourlanos et al. developed a screening tool to identify autoimmune diabetes by integrating the clinical features of age of onset, acute symptoms, body mass index (BMI), and personal or family history of autoimmune disease (8).

Chinese patients with diabetes have somewhat different clinical characteristics compared with other populations. Most new cases of T1DM in China are adults, and Chinese individuals with T2DM are generally less obese than their Caucasian counterparts (9). However, there are no clinical diagnostic models of T1DM in Chinese patients with new-onset diabetes. Thus, in this study, we used the eXtreme Gradient Boosting (XGBoost) algorithm to generate several machine learning models according to the clinical features of participants. The purpose of the algorithm was to identify an optimal diagnostic model to distinguish T1DM from T2DM in adults newly diagnosed with diabetes. We present the following article in accordance with the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting checklist (available at http://dx.doi.org/10.21037/atm-20-7115).

Methods

Study population

We analyzed data from 18,891 participants with newly diagnosed diabetes from a nationwide, multi-center, cross-sectional survey performed from April 2015 to October 2017. Forty-six tertiary care hospitals were invited from 20 provinces and 4 municipalities, across all 7 geographic regions of China (4 Northeast, 8 North, 3 Northwest, 9 Central, 3 Southwest, 7 South, and 12 East). The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The Ethics Committees of the Second Xiangya hospitals, Central South University in China approved this study (No. 2014032), and all patients provided written informed consents.

The following factors were inclusion criteria: (I) diagnosis of diabetes that met the World Health Organization 1999 criteria (10); (II) age 18 years and older; (III) diabetes duration less than 1 year; and (IV) outpatients attending clinics in the department of endocrinology. Individuals were excluded if pregnant at the time of diabetes diagnosis, if they had gestational diabetes mellitus, or if they had co-existing acute diseases such as infection or acute myocardial infarction that could affect glucose metabolism. In addition, we excluded 1,844 patients who lacked data on key variables for diabetes classification and 1,841 patients who did not meet the criteria for identifying types of diabetes (shown in model outcome part below). We analyzed the data of the remaining 15,206 patients for model construction ().

Figure 1

Flow diagram of the study design. GADA, glutamic acid decarboxylase autoantibody; T1DM, type 1 diabetes; T2DM, type 2 diabetes.

Clinical measurements and data collection

Research nurses at each of the 46 participating hospitals participated in a series of training programs to standardize all procedures and methods of data collection. Patients self-reported demographic characteristics (i.e., age and sex), clinical features, and lifestyle risk factors (i.e., exercise habits, diet, smoking, and alcohol consumption). The nurses used standard procedures to measure patient height, weight, waist circumference, hip circumference, and blood pressure.

Laboratory assays

Fasting plasma glucose (FPG), total cholesterol (TC), triglycerides (TGs), high-density lipoprotein cholesterol (HDL-C), low-density lipoprotein cholesterol (LDL-C), fasting C-peptide (FCP), and plasma hemoglobin A1c (HbA1c) were assayed by standard methods at the study sites. Postprandial blood samples were tested for 2-h postprandial plasma glucose (PPG) and C-peptide. Serum samples for GADA assays were shipped on ice within 1 day and stored at –80 °C. The core laboratory (Central South University) performed serum GADA assays by a standardized radioligand assay as reported (11). The assay was assessed in the 2016 Islet Autoantibody Standardization Program (IASP 2016).

Model outcome: identification of T1DM and T2DM

T1DM was defined as postprandial C-peptide (PCP) less than 0.3 nmol/L or GADA titer no less than 173.5 U/mL. T2DM was defined as PCP more than 0.6 nmol/L and GADA titer less than 18.2 U/mL.

Statistical analysis

Continuous variables were expressed as mean [standard deviation (SD)] or median (interquartile range) based on evaluation of normal distribution; categorical variables were given as number (percent). For analysis of continuous variables, t-test or Mann-Whitney test were performed to compare differences between groups where appropriate. Frequency differences were compared using Chi-square test. P<0.05 was considered statistically significant.

Model development using XGBoost

XGBoost is a machine learning method for classification problems (12). XGBoost produces classification models in the form of ensembles of weak classification models, typically decision trees. XGBoost can provide lower bias and better optimize an objective function, compared with traditional linear methods, e.g., logistic regression. We used XGBoost to build a multivariable clinical diagnostic model to identify T1DM and T2DM.

Importance score of variables

For each variable, XGBoost provides an importance score that represents the variable’s contribution to predict the class label in the model. The larger the feature score of a variable, the more important is the variable to the model. In our study, we chose the method “gain”, which is the average gain of the splits that use the variable, to compute the importance score of each variable.

First, we used all candidate variables to build the initial XGBoost model to assist in distinguishing T1DM and T2DM. Then, we obtained the importance score of each variable and ranked the variables according to those scores. We selected the most important variables from the candidate variables to build the final model intended for actual clinical practice.

Missing values

XGBoost can automatically accommodate missing values, i.e., by using a default direction for the missing values in each tree node (12). If a missing value appears in the validation dataset, it will be handled automatically by following the default direction that is decided in the training phase.

Explanation of classification results

To interpret the model output, we used the SHapley Additive exPlanations (SHAP) method to explain the XGBoost classification results (13). By using the SHAP method, we transformed the XGBoost model to the accumulative effects of all variable attributions on the output probability of diabetes type for each patient. In this manner, the impact of the variables on the outcome from the SHAP transformation can be interpreted easily. A SHAP value of a variable represents its impact on the model output.

Evaluation and comparisons of model performance

We split the dataset randomly into a training set (80%) and a testing set (20%). The training set was used to develop models. For validation, we used the testing set to assess the performance of the models. We computed area under the receiver operating characteristic curve (ROC AUC) of each model. We also computed model performance at different cut-offs for different models. The model performance metrics were sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and Youden’s index. We developed models with different combinations of variables and selected the final model by comparing the performance of these models. We also performed a 5-fold cross validation to further assess the validity of these models. Basically, we conducted the model development using 4 folds and the model validation using the left fold. Each fold was included in the training set 4 times and in the testing set 1 time.

In order to evaluate the agreement of observed outcomes and predictions of the final model, we discretized the prediction space into 10 bins and draw a calibration curve by plotting T1DM diagnostic predictions on the X-axis and observed frequency of T1DM on the Y-axis and presented the intercept and slope for this curve. An optimal calibration was represented with an intercept of 0 and a slope of 1.

Results

Characteristics of the study participants

Among the 15,206 patients with data for all key variables, 1,465 (9.63%) patients had T1DM and 13,741 (90.37%) had T2DM. The average age of disease onset was 43.7±15.1 and 51.0±12.8 years in the T1DM and T2DM group, respectively. As expected, patients with T1DM were significantly leaner, and they had lower blood pressure and better lipid metabolic parameters but higher serum FPG and HbA1c concentrations. Lifestyle factors liking drinking, diet treatment and physical activity were more frequent in patients with T2DM than in those with T1DM ().

Table 1

Characteristics of patients diagnosed with T1DM and T2DM

Variables	T1DM			T2DM		P value
	N	Value		N	Value
Age of onset (years)	1,465	43.7±15.1		13,741	51.0±12.8	<0.001
Sex						0.227
Female (n, %)	567	38.7%		5,542	40.3%
Male (n, %)	898	61.30%		8,199	59.67%
BMI (kg/m2)	1,416	21.5±3.5		13,385	25.1±3.5	<0.001
FPG (mmol/L)	1,419	9.33 (6.60, 13.43)		13,603	7.87 (6.52, 10.31)	<0.001
PPG (mmol/L)	1,397	17.0±7.1		13,462	15.1±5.5	<0.001
HbA1c (%)	1,417	11.6 (9.0, 13.6)		13,491	8.6 (6.9, 10.9)	<0.001
SBP (mmHg)	1,415	120.5±15.6		13,139	128.2±16.2	<0.001
DBP (mmHg)	1,417	76.2±10.4		13,139	80.4±10.5	<0.001
Waist (cm)	1,320	80.6±10.3		12,478	89.1±10.5	<0.001
TG (mmol/L)	1,419	1.2 (0.8, 1.8)		13,262	1.77 (1.22, 2.75)	<0.001
TC (mmol/L)	1,422	4.6±1.4		13,366	4.8±1.3	<0.001
LDL-C (mmol/L)	1,421	2.7±1.0		13,327	2.9±1.0	<0.001
HDL-C (mmol/L)	1,413	1.2 (1.0, 1.5)		13,205	1.11 (0.94, 1.32)	<0.001
FCP (nmol/L)	1,456	0.14 (0.06, 0.24)		13,576	0.62 (0.43, 0.85)	<0.001
PCP (nmol/L)	1,465	0.22 (0.12, 0.34)		13,741	1.60 (1.09, 2.40)	<0.001
Current smoking (n, %)	448	31.0%		4,084	30.1%	0.485
Current drinking (n, %)	212	14.8%		2,447	18.1%	0.002
Diet treatment (n, %)	687	53.5%		6,240	62.3%	<0.001
Physical activity (n, %)	568	44.2%		5,357	53.5%	<0.001

% reported for all categorical variables. Data are presented as mean ± SD or median with upper and lower quartiles based on evaluation of normal distribution. Differences between T1DM and T2DM were compared using t-test or Mann-Whitney test for continuous variables where appropriate and chi-square test for categorical variables. T1DM, type 1 diabetes; T2DM, type 2 diabetes; BMI, body mass index; FPG, fasting plasma glucose; PPG, postprandial plasma glucose; HbA1c, hemoglobin A1c; SBP, systolic blood pressure; DBP, diastolic blood pressure; TG, triglyceride; TC, total cholesterol; LDL-C, low-density lipoprotein cholesterol; HDL-C, high-density lipoprotein cholesterol; FCP, fasting C-peptide; PCP, postprandial C-peptide.

Classification model construction for identifying T1DM and T2DM

An XGBoost classification model was used to identify T1DM and T2DM. We used 13 clinical features to distinguish patients. The features were ordered by the importance scores of each value in this model (). BMI contributed the most to the model, followed by age of onset, HbA1c, waist dimension, TG concentration, and FPG.

Figure 2

Feature importance scores of clinical variables. BMI, body mass index; HbA1c, hemoglobin A1c; TG, triglyceride; FPG, fasting plasma glucose; PPG, postprandial plasma glucose; SBP, systolic blood pressure; DBP, diastolic blood pressure; HDL-C, high-density lipoprotein cholesterol; TC, total cholesterol; LDL-C, low-density lipoprotein cholesterol.

We used SHAP values, which referred to the impact of one variable on model output, to explain the classification results. presented how the contribution of an individual variable on the model output was affected by its value. The y position of the dot was the impact of each variable on the diagnosis of T1DM, which can be also interpreted as an increase (above 0) or decrease (below 0) of probability of being diagnosed as T1DM. From , the SHAP value decreased as the age of onset grew and the age of onset was about 40 years when SHAP value was 0, which indicated that an age of onset less than 40 increased the probability of having T1DM (). Other risk factors that increased the probability of being diagnosed with T1DM included a BMI lower than 23 kg/m2 (), HbA1c higher than 7% (), and waist circumference lower than 85 cm (). Among four lipid parameters, only TG influenced the probability of being identified as T1DM. Patients with TG less than about 1.8 mmol/L () had a higher probability of being identified as T1DM, although there was no obvious association between the level of TC, LDL-C, and HDL-C () and the probability for identifying as T1DM.

Figure 3

Impact of (A) age at onset, (B) BMI, (C) HbA1c, (D) waist, (E) TG, (F) TC, (G) LDL-C, (H) HDL-C on SHAP value for test dataset only. SHAP value represents the impact of each variable on the model output (diagnosis of T1DM in this model). SHAP, Shapley Additive exPlanations; BMI, body mass index; HbA1c, hemoglobin A1c; TG, triglyceride; TC, total cholesterol; LDL-C, low-density lipoprotein cholesterol; HDL-C, high-density lipoprotein cholesterol; T1DM, type 1 diabetes.

Classification model evaluation for identifying T1DM and T2DM

To make the model concise and practical, we attempted to limit the number of input variables without significantly losing model performance. Because BMI and age of onset were easy to acquire and they ranked high by feature importance score, we included them in the final model. We set the input variables as the combination of BMI, age of onset, and one other variable, and built the candidate models accordingly. We also built models with all 13 variables and a model with only BMI and age of onset. We presented the corresponding cut-off, specificity, and sensitivity, when the Youden’s index was the highest. Then, we compared the ROC AUC, sensitivity, specificity, PPV, NPV, and Youden’s index for all the models. Finally, to further assess the validity of these models, a 5-fold cross validation was performed and the average ROC AUC, sensitivity, specificity, PPV, and NPV for each model were evaluated.

We ranked the model performance results according to their ROC AUC. The ROC AUC of the model with all 13 features was 0.86 (95% CI, 0.83 to 0.88) and the sensitivity and specificity of this model were both 0.78 (). The ROC AUC of the model that used only BMI and age of onset was 0.80 (95% CI, 0.77 to 0.83) and the sensitivity, specificity, PPV, and NPV were 0.61, 0.86, 0.32, and 0.95.

Table 2

Performance of models with different combinations of variables

Features	ROC AUC (95% CI)	Cut-off (%)	Youden’s index	Sensitivity	Specificity	PPV	NPV
All 13 features	0.86 (0.83, 0.88)	7	0.56	0.78	0.78	0.27	0.97
BMI + age of onset + HbA1c	0.83 (0.80, 0.85)	8	0.52	0.77	0.76	0.25	0.97
BMI + age of onset + FPG	0.82 (0.78, 0.84)	9	0.49	0.71	0.77	0.25	0.96
BMI + age of onset + TG	0.81 (0.78, 0.84)	8	0.48	0.73	0.75	0.23	0.96
BMI + age of onset + PPG	0.81 (0.78, 0.84)	10	0.49	0.69	0.80	0.27	0.96
BMI + age of onset + men	0.80 (0.77, 0.83)	14	0.48	0.60	0.88	0.34	0.95
BMI + age of onset + TC	0.80 (0.77, 0.83)	10	0.48	0.67	0.81	0.27	0.96
BMI + age of onset + HDL-C	0.80 (0.77, 0.83)	12	0.46	0.61	0.85	0.30	0.95
BMI + age of onset + DBP	0.80 (0.77, 0.83)	13	0.46	0.61	0.86	0.31	0.95
BMI + age of onset + LDL-C	0.80 (0.77, 0.83)	14	0.47	0.60	0.87	0.33	0.95
BMI + age of onset + SBP	0.80 (0.77, 0.83)	9	0.47	0.69	0.77	0.24	0.96
BMI + age of onset	0.80 (0.77, 0.83)	13	0.47	0.61	0.86	0.32	0.95
BMI + age of onset + waist	0.79 (0.76, 0.82)	10	0.46	0.67	0.79	0.25	0.96

ROC AUC, area under the receiver operating characteristic curve; PPV, positive predictive value; NPV, negative predictive value; BMI, body mass index; HbA1c, hemoglobin A1c; FPG, fasting plasma glucose; TG, triglyceride; PPG, postprandial plasma glucose; TC, total cholesterol; HDL-C, high-density lipoprotein cholesterol; DBP, diastolic blood pressure; LDL-C, low-density lipoprotein cholesterol; SBP, systolic blood pressure.

When we used only three variables to build models, the highest ROC AUC model was a combination of BMI, age of onset, and HbA1c. The ROC AUC, sensitivity, specificity, PPV, and NPV of this model were 0.83 (95% CI, 0.80 to 0.85), 0.77, 0.76, 0.25, and 0.97, respectively. This model was followed by a model composed of BMI, age at onset, and FPG, a model composed with BMI, age at onset, and TG, and a model with BMI, age of onset, and PPG; the ROC AUCs for all these models were from 0.81 (95% CI, 0.78 to 0.84) to 0.82 (95% CI, 0.78 to 0.84) (). All these models had similar discriminatory performances when cross validation was performed (Table S1).

Calibration of the final diagnostic model

We chose the model composed of age of onset, BMI, and HbA1c as the final model since it had the highest discriminatory performance among all models with less variables in .

Calibration evaluation was assessed for the final model in the testing dataset. A calibration curve for the final model was plotted and the intercept and slope for this curve were 0.02 (95% CI, –0.03 to 0.06) and 0.90 (95% CI, 0.79 to 1.02), respectively, suggesting that this model did well at identifying T1DM cases ().

Figure 4

Calibration curve of the model composed of BMI, age of onset and HbA1c. BMI, body mass index; HbA1c, hemoglobin A1c.

Online classification tool

Based on the final model composed of age of onset, BMI, and HbA1c, we designed an online classification tool that can compute the probability of T1DM or T2DM based on the input values (available at http://cdss.pingan.com:8082/diabetes/index.html).

Discussion

We established a diagnostic model of high performance to identify patients with newly diagnosed diabetes as likely to have T1DM.

The model was composed of age of onset, BMI, and HbA1c parameters, all easily accessed data. These parameters are associated with beta cell function and disease progression in T1DM. Early age at diagnosis of T1DM was associated with more rapid decline in beta-cell function (14,15). The association between BMI and T1DM progression is conflicting. Some studies showed that patients with higher BMI had better beta cell function at diagnosis and after follow-up (16-18). However, other studies showed that BMI might be a risk factor for developing T1DM and for accelerating T1DM progression (19-21). Better HbA1c levels were associated with higher C-peptide concentrations at diagnosis and could predict residual beta cell function and long-term diabetes control after follow-up (22-24).

Our study has multiple strengths. Previous studies on distinguishing T1DM from T2DM did not focus on patients with new-onset diabetes, whereas the participants in our study had short disease durations of less than 1 year (25-28). Patients with T1DM can obtain timely insulin treatment if they are diagnosed early and correctly. Additionally, diagnosis criteria in our study were based on C-peptide and GADA concentrations. GADA present at early preclinical stages of T1DM and the titer of GADA are important for treatment decision and disease progression in patients with latent autoimmune diabetes in adults (LADA) (29-33). Patients with LADA are defined by adult age of onset, insulin independence for at least 6 months after diagnosis, and positivity for islet autoantibodies. They present a similar clinical characteristic to T2DM at the onset of disease and require insulin treatment in the early phase. In our study, GADA-positive patients with relatively good beta cell function were also included as T1DM; thus, our algorithm could also identify individuals with LADA.

Our study had limitations. We did not include patients with gestational diabetes mellitus or maturity-onset diabetes of the young. Patients with gestational diabetes are relatively easy to identify, and often they are diagnosed in an obstetrical department instead of an endocrinology department. Patients with maturity-onset diabetes usually do not have beta cell failure or GADA positivity; thus, there is a low possibility for misdiagnosis of these individuals as having T1DM. The possibility of being diagnosed with T1DM could be overestimated due to the suppression of beta cell function when beta cells are chronically exposed to hyperglycemia, an overlap C-peptides concentration with some subjects with T2DM (34). However, our goal is to decrease the misdiagnosis rate of T1DM by identifying cases who are likely to be diagnosed with T1DM and treat them with insulin timely. By providing beta cell repose in the early phase of the disease, insulin treatment may be beneficial for patients misdiagnosed with T1DM.

In conclusion, we show that a diagnostic model that integrates age of onset, BMI, and HbA1c could distinguish T1DM from T2DM among adult patients with newly diagnosed diabetes, which provide a useful tool in assisting physicians in subtyping and precising treatment in diabetes.

The article’s supplementary files as