Development of a predictive risk model for all-cause mortality in patients with diabetes in Hong Kong.

INTRODUCTION: Patients with diabetes mellitus are risk of premature death. In this study, we developed a machine learning-driven predictive risk model for all-cause mortality among patients with type 2 diabetes mellitus using multiparametric approach with data from different domains. RESEARCH DESIGN AND METHODS: This study used territory-wide data of patients with type 2 diabetes attending public hospitals or their associated ambulatory/outpatient facilities in Hong Kong between January 1, 2009 and December 31, 2009. The primary outcome is all-cause mortality. The association of risk variables and all-cause mortality was assessed using Cox proportional hazards models. Machine and deep learning approaches were used to improve overall survival prediction and were evaluated with fivefold cross validation method.
RESULTS: A total of 273 678 patients (mean age: 65.4±12.7 years, male: 48.2%, median follow-up: 142 (IQR=106-142) months) were included, with 91 155 deaths occurring on follow-up (33.3%; annualized mortality rate: 3.4%/year; 2.7 million patient-years). Multivariate Cox regression found the following significant predictors of all-cause mortality: age, male gender, baseline comorbidities, anemia, mean values of neutrophil-to-lymphocyte ratio, high-density lipoprotein-cholesterol, total cholesterol, triglyceride, HbA1c and fasting blood glucose (FBG), measures of variability of both HbA1c and FBG. The above parameters were incorporated into a score-based predictive risk model that had a c-statistic of 0.73 (95% CI 0.66 to 0.77), which was improved to 0.86 (0.81 to 0.90) and 0.87 (0.84 to 0.91) using random survival forests and deep survival learning models, respectively.
CONCLUSIONS: A multiparametric model incorporating variables from different domains predicted all-cause mortality accurately in type 2 diabetes mellitus. The predictive and modeling capabilities of machine/deep learning survival analysis achieved more accurate predictions. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY. Published by BMJ.

Increased variability in metabolic parameters is predictive of higher mortality in type 2 diabetes mellitus.

We developed a machine learning-driven predictive risk model for type 2 diabetes mellitus using multiparametric approach with data from different domains.

Measures of variability of fasting glucose and HbA1c show similar predictive power for all-cause mortality, regardless of whether adjustments were made for initial values or mean values across follow-up.

A multiparametric predictive risk model incorporating variables from different domains, including baseline demographics, comorbidities and laboratory tests, measures of variability of HbA1c and fasting blood glucose predicted all-cause mortality accurately.

Machine learning-driven algorithms further improved the accuracy of the predictive models.

A simple, easy-to-use score-based system has been devised to enable rapid risk prediction in the clinical setting.

Introduction

Type 2 diabetes mellitus is one of the most common metabolic conditions, with an increasing prevalence attributable to aging, sedentary lifestyles, environmental changes and better disease management.1–3 Patients with this condition are at an increased risk of premature death and other complications.4 5 Existing risk models have been developed, such as QDiabetes for predicting new onset diabetes,6 and CORE,7 BRAVO8 and Michigan9 models for predicting disease progression, complications and mortality. These have generated good predictive results in western cohorts but are limited by their direct applicability to Asian populations. For example, Chinese patients have a lower body mass index threshold for diabetes development and have a higher propensity to suffer from chronic kidney disease as a result.10 11 While Asian population-specific models are available,12–15 these have generally not incorporated temporal measures of variability for longitudinal data or machine learning approaches, both of which can enhance risk prediction.16 17 Indeed, with the rapid development of big data analytics, it has become easier to improve discrimination by analyzing complex interactions among variables. Previously, a machine learning-driven approach has demonstrated superior performance for predicting diabetes onset in a Chinese cohort.18

In this territory-wide study, with the aid of machine/deep learning approaches, we developed a risk model for mortality prediction using multiparametric data from different domains. These include baseline comorbidities, measures of variability of fasting glucose and HbA1c, inflammatory and nutritional indices and drug prescription details. We tested the hypothesis that machine learning methods (random survival forests, RSF19) and deep neural survival learning models (DeepSurv20) can significantly improve predictive performance when compared with Cox regression-based models.

Methods

Study design and data source

The inclusion criteria were patients who received antidiabetic medications or had International Classification of Disease, Ninth Edition (ICD-9) codes for type 2 diabetes mellitus, and attended any of the 43 public hospitals or their associated ambulatory or outpatient facilities managed by the Hong Kong Hospital Authority between January 1 and December 31, 2009. The Clinical Data Analysis and Reporting System, a healthcare database that integrates patient information to establish comprehensive medical records with accurately linked mortality data were used in this study. This system has been used for epidemiological research by multiple research teams, including our team, in the past,21–24 including model development studies.25 26

Data extraction

Baseline patient characteristics include demographic details such as age and sex, prior comorbidities (heart failure (HF), ischemic heart disease (IHD), ischemic stroke, aborted sudden cardiac death (SCD) of all causes including acute myocardial infarction, atrial fibrillation (AF), peripheral vascular disease, intracranial hemorrhage, osteoporosis, dementia, hypertension, chronic obstructive pulmonary disease (COPD), cancer, renal and ophthalmological diabetic complications), antidiabetic and cardiovascular medications. The ICD-9 codes for the aforementioned comorbidities are summarized in online supplemental table 1. The duration of living with type 2 diabetes mellitus from the point of diagnosis until December 31, 2009 was also extracted, and determined by the earliest fulfillment of any of the following criteria in this order: (1) initial documentation of type 2 diabetes mellitus related ICD-9 codes; (2) earliest HbA1c>6.5%; (3) earliest fasting blood glucose (FBG) >7 mmol/L. Time-till all-cause mortality was determined as the number of days from the starting date of patient inclusion, January 1, 2009, until the day of death or the end of the follow-up period, December 31, 2019.

The following laboratory data were collected at baseline: neutrophil-lymphocyte ratio (NLR) was derived by dividing the absolute neutrophil by the lymphocyte count, anemia defined as <13 g/dL for men and <12 g/dL for women, biochemical test results including (1) creatinine, sodium, potassium, (2) urea, (3) albumin and total protein, (4) alanine aminotransferase and alkaline phosphatase, (5) FBG and HbA1c; (6) high-density lipoprotein-cholesterol (HDL-C), directly measured low-density lipoprotein-cholesterol (LDL-C), total cholesterol, and triglyceride.

The number of antidiabetic drugs by class were extracted: (1) insulin, (2) biguanide, (3) sulphonylurea, (4) alpha-glucosidase inhibitor, (5) thiazolidinedione, (6) dipeptidyl peptidase-4 inhibitor, (7) glucagon-like peptide receptor-1 agonist, (8) meglitinide. Similarly, the number of antihypertensive medications of the following classes were also extracted: (1) angiotensinogen-converting-enzyme inhibitor/angiotensin receptor blocker, (2) beta-adrenergic receptor blocker, (3) calcium channel blocker, (4) diuretics. Lipid-lowering agents were also extracted.

Variability calculations

To calculate FBG and HbA1c variability, data points were obtained for the period between January 1, 2004 and December 31, 2008. Only patients with three or more measurements for the specific parameter were included for the variability analysis of the respective parameter. The different measures are detailed below and summarized in online supplemental table 2: (1)SD, (2) absolute variability score defined as 100× no. of measurements>0.5/no. of measurements, (3) percentage variability score defined as 100× no. of measurements>10% of previous measurement/no. of measurements, (4) normalized absolute variability score given by (2)/individual mean, (5) normalized percentage variability score given by (3) /individual mean, (6) SD/individual baseline, (7) coefficient of variation given by SD/individual mean, (8) variability independent of mean given by SD/individual meanˆ(ln(population SD)/ln(population mean)).

Outcomes and statistical analysis

The primary outcome for the present student is all-cause mortality. Univariate Cox regression was applied to identify significant predictors for all-cause mortality and HR with 95% CI were reported. Variables achieving p<0.10 were included in a diabetes duration-adjusted multivariate model. Statistical significance is defined as p<0.05. FBG and HbA1c variability of the same formula were paired and added to the multivariate model to assess their predictiveness through comparison of HR, thus preventing problems with collinearity.

To generate a predictive score, Cox regression was repeated for the final multivariate model with measures of variability included. HR between 1 and 1.50 was awarded 1 mark in the score. To adjust for the U-shaped relationship against mortality reported for HDL-C, LDL-C, total cholesterol and HbA1c, these parameters were first divided by deciles to serve as cut-offs and undergo univariate Cox regression. Thereafter, the decile with the smallest HR was selected as the reference and compared against the remaining deciles through univariate Cox regression again. The minimum and maximum cut-offs for the deciles that had an insignificant difference with the reference decile were selected as the cut-offs to be used in the score. To demonstrate the U-shaped relationship, the HR of deciles was plotted graphically. Similar methods were employed to illustrate the U-shaped relationship by existing studies.27–29 Cut-off values for continuous variables in the score were found through maximizing sensitivity and specificity. Age and diabetes duration were rounded to the nearest whole number, while other parameters were rounded to two decimal points. The predictive value of the score was evaluated through the generation of a receiver operating characteristic (ROC) curve and area under the-curve (AUC) calculated.

To further evaluate the predictive value of the measures of variability, the measures were also divided into quartiles, with the first quartile as a reference, to perform univariate Cox regression and assess the AUC of the quartile cut-offs. The quartile HR of the FBG and HbA1c measures of variability were illustrated graphically. Statistical analyses were performed using RStudio software (V.1.1.456) and Python (V.3.6).

Development of machine/deep models for survival learning

Machine/deep learning survival analysis models can directly capture the relationships between risk predictors and mortality outcome without prior functional assumptions typically made in Cox analysis models. Here we used an RSF model, a type of machine learning method for survival analysis, relying on the intuition that the best survival learning model, when combined with weak decision tree learning models, can minimize the overall survival prediction errors. The prediction errors are measured by performance evaluators, for example, precision, recall, AUC and C-index. The out-of-bag (OOB) method was adopted whenever a bootstrap sample (bag ones) is down with replacement from the training dataset. The bootstrapping technique is used to grow the tree and results in well-defined subsets. Some of the bootstrap are duplicates and are members of the in-bag subset, and the remaining individuals define the OOB subset for the final tree. Each individual in the OOB subset for a tree is passive. A unique terminal node membership and terminal node statistic were assigned. An OOB ensemble statistic for each individual is formed by combining the terminal node statistics from all trees where an individual is an OOB member. Finally, the class with the maximum frequency in the OOB ensemble statistic serves as the predicted class label for the member. More detailed descriptions of these concepts were described by Breiman et al.30

The variable’s importance of interest is calculated as the prediction error (squared loss) of the original ensemble event-specific cumulative probability function subtracted from the prediction error of the original ensemble event-specific cumulative probability function (obtained when each OOB instance is just dropped down its in-bag competing risks tree).31 32 In this study, RSF was used for mortality prediction and the most important predictors were ranked according to variable importance measure in RSF. Variables that were important predictors of risk outcome have a larger importance value, indicating higher predictive strength, whereas non-predictive variables have zero or negative values.

We further employed a nonlinear deep learning survival method termed Cox proportional hazards DeepSurv approach. This can inherently and adaptively model the high-level interaction patterns among risk predictors and thus can better capture the complex nonlinear relationship between patients’ covariates (eg, clinical features) and mortality outcome directly. Specifically, DeepSurv is a deep feed-forward neural network that can predict the effects of a patient’s baseline covariates on their hazard rate parameterized by the weights of the neural network. The input of DeepSurv is the baseline variables of the patient with diabetes. The hidden layers of DeepSurv consist of a fully connected layer of nodes, followed by a dropout layer.33 The output of the DeepSurv is a single node with a linear activation which estimates the log-risk function in the Cox model. In this study, we train DeepSurv by presetting the objective function to be the average negative log form of Cox partial likelihood with L2-regularization,34 in order to model for mortality risk prediction of the patients with diabetes. Gradient descent optimization was used to find the weights of DeepSurv. The hyperparameters of DeepSurv including the number of hidden layers, the number of nodes in each layer and dropout probability were determined from a random hyperparameter search approach.35

The RSF, DeepSurv and multivariate Cox regression models adopted the same set of predictors. A fivefold cross-validation approach was performed to compare the survival prediction performance of RSF and DeepSurv in terms of precision, recall, AUC and HC-index over the standard Cox model. The R packages, randomForestSRC (V.2.9.3), ggplot2 (V.3.3.2), and python package DeepSurv (V.0.1.0) were used to generate the mortality prediction results.

Results

Baseline characteristics

The study cohort included 273 876 patients (mean age: 65.4±12.7 years, male: 48.2%, diabetes duration=6.18 ± 4.56 years) with a median follow-up of 142 (IQR (IQR)=106–142) months, which corresponded to a total of 2 660 465 patient-years. The baseline demographics, clinical, laboratory and drug details are shown in tables 1 and 2 for continuous and discrete variables, respectively. The most prevalent comorbidities were hypertension, IHD and HF. The percentage of patients on n=0, 1, 2, 3, and 4 antidiabetic medications were 13.3%, 34.8%, 46.1%, 5.4% and 0.4%, respectively. At baseline, the fasting glucose and HbA1c were 8.02±1.95 mmol/L and 7.75%±2.59%, respectively. The median number for fasting glucose and HbA1c measurements were 7 (IQR=4–11) and 7 (IQR=4–10), respectively. The different measures of variability for fasting glucose or HbA1c are quantified for subsequent use to predict mortality (detailed methodology is shown in online supplemental table 2.

Predictors of all-cause mortality

Over a median follow-up period of 142 (IQR=106–142) months, 91 155 deaths were recorded (33.3%), which corresponded to an annualized mortality rate of 3.43%. The significant univariate predictors for all-cause mortality are presented in table 3. All measures of variability for FBG and HbA1c were significant predictors as well. The graphical comparison of HR from quartile cut-offs of FBG and HbA1c variability predictors are shown in online supplemental figure 1A, B, with the details summarized in online supplemental tables 3 and 4.

Table 3

Univariate predictors for all-cause mortality

	HR	95% CI	P value
Age	1.090	(1.089 to 1.091)	<0.0001
Male	1.12	(1.11 to 1.14)	<0.0001
Complete blood count
Neutrophil-lymphocyte ratio	1.033	(1.032 to 1.034)	<0.0001
Baseline anemia	3.50	(3.45 to 3.55)	<0.0001
Lipid profile
HDL-C	0.836	(0.815 to 0.857)	<0.0001
LDL-C	0.883	(0.874 to 0.892)	<0.0001
Total cholesterol	0.910	(0.903 to 0.916)	<0.0001
Triglyceride	0.963	(0.957 to 0.970)	<0.0001
Comorbidity
Renal diabetic complication	3.68	(3.54 to 3.83)	<0.0001
Ophthalmological diabetic complication	2.73	(2.62 to 2.84)	<0.0001
Peripheral vascular disease	4.39	(3.91 to 4.93)	<0.0001
Ischemic stroke	2.85	(2.78 to 2.93)	<0.0001
Sudden cardiac death	2.48	(2.40 to 2.56)	<0.0001
Atrial fibrillation	3.54	(3.45 to 3.64)	<0.0001
Heart failure	4.74	(4.64 to 4.85)	<0.0001
Intracranial hemorrhage	2.70	(2.59 to 2.82)	<0.0001
Ischemic heart disease	2.24	(2.20 to 2.28)	<0.0001
Osteoporosis	2.87	(2.34 to 3.52)	<0.0001
Dementia	5.92	(5.69 to 6.16)	<0.0001
Hypertension	2.55	(2.52 to 2.59)	<0.0001
Chronic obstructive pulmonary disease	4.55	(4.22 to 4.91)	<0.0001
Cancer	2.48	(2.42 to 2.54)	<0.0001
FBG
Mean	1.00	(0.997 to 1.01)	0.527
Absolute successive variability score	1.008	(1.007 to 1.008)	<0.0001
Percentage successive variability score	1.01	(1.009 to 1.01)	<0.0001
SD	1.15	(1.15 to 1.16)	<0.0001
Normalized absolute successive variability score	1.065	(1.06 to 1.07)	<0.0001
Normalized percentage successive variability score	1.07	(1.067 to 1.074)	<0.0001
SD/initial FBG	1.01	(1.009 to 1.01)	<0.0001
Coefficient of variation	1.019	(1.018 to 1.019)	<0.0001
Variability independent of mean	1.011	(1.01 to 1.011)	<0.0001
HbA1c
Mean	1.07	(1.06 to 1.07)	<0.0001
Absolute successive variability score	1.01	(1.007 to 1.01)	<0.0001
Percentage successive variability score	1.008	(1.08 to 1.009)	<0.0001
SD	1.19	(1.18 to 1.20)	<0.0001
Normalized absolute successive variability score	1.055	(1.05 to 1.06)	<0.0001
Normalized percentage successive variability score	1.063	(1.06 to 1.07)	<0.0001
SD/initial HbA1c	1.014	(1.01 to 1.014)	<0.0001
Coefficient of variation	1.017	(1.016 to 1.018)	<0.0001
Variability independent of mean	1.011	(1.01 to 1.012)	<0.0001

FBG, fasting blood glucose; HDL-C, high-density lipoprotein-cholesterol; LDL-C, low-density lipoprotein cholesterol.

The following parameters remained significant predictors following multivariate adjustment (table 4): (1) age and male gender, baseline comorbidities or complications (hypertension, HF and AF, COPD, cancer, dementia, ischemic stroke, intracranial hemorrhage, aborted SCD, diabetic renal and ophthalmological complications), (2) laboratory tests (anemia, neutrophil-to-lymphocyte ratio (NLR); HDL-C, total cholesterol, triglyceride; mean HbA1c and mean FBG), (3) eight different measures of variability for HbA1c and FBG (table 4). A U-shaped relationship between HDL-C, LDL-C, total cholesterol (figure 1A–C), but not for triglyceride (figure 1D) and all-cause mortality. A U-shaped relationship was also observed for HbA1c but not for FBG (figure 1E, F).

Table 4

Multivariate Cox regression for all-cause mortality

	HR	95% CI	P value
Age	1.06	(1.06 to 1.06)	<0.0001
Male	1.35	(1.31 to 1.40)	<0.0001
Complete blood count
Neutrophil-lymphocyte ratio	1.02	(1.02 to 1.03)	<0.0001
Baseline anemia	1.94	(1.87 to 2.01)	<0.0001
Lipid profile
High-density lipoprotein cholesterol	0.891	(0.849 to 0.935)	<0.0001
Low-density lipoprotein cholesterol	1.01	(0.986 to 1.04)	0.348
Total cholesterol	1.04	(1.01 to 1.06)	0.001
Triglyceride	1.02	(1.01 to 1.03)	0.001
Comorbidity
Renal diabetic complication	1.28	(1.20 to 1.36)	<0.0001
Ophthalmological diabetic complication	1.18	(1.11 to 1.26)	<0.0001
Peripheral vascular disease	1.16	(0.984 to 1.37)	0.078
Ischemic stroke	1.25	(1.18 to 1.32)	<0.0001
Sudden cardiac death	1.17	(1.09 to 1.25)	<0.0001
Atrial fibrillation	1.30	(1.23 to 1.37)	<0.0001
Heart failure	1.62	(1.54 to 1.69)	<0.0001
Intracranial hemorrhage	1.28	(1.16 to 1.41)	<0.0001
Ischemic heart disease	1.01	(0.971 to 1.05)	0.574
Osteoporosis	1.03	(0.769 to 1.38)	0.842
Dementia	1.81	(1.64 to 2.00)	<0.0001
Hypertension	1.30	(1.26 to 1.35)	<0.0001
Chronic obstructive pulmonary disease	1.43	(1.20 to 1.70)	<0.0001
Cancer	1.41	(1.33 to 1.49)	<0.0001
Mean FBG	1.01	(1.00 to 1.02)	0.011
Mean HbA1c	1.06	(1.04 to 1.08)	<0.0001
FBG
Absolute successive variability score	1.00	(1.00 to 1.00)	0.033
Percentage successive variability score	1.00	(1.00 to 1.00)	<0.0001
SD	1.08	(1.07 to 1.10)	<0.0001
Normalized absolute successive variability score	1.02	(1.01 to 1.02)	<0.0001
Normalized percentage successive variability score	1.03	(1.02 to 1.03)	<0.0001
SD/initial	1.00	(1.00 to 1.00)	<0.0001
Coefficient of variation	1.01	(1.01 to 1.01)	<0.0001
Variability independent of mean	1.01	(1.01 to 1.01)	<0.0001
HbA1c
Absolute successive variability score	1.00	(1.00 to 1.00)	<0.0001
Percentage successive variability score	1.00	(1.00 to 1.00)	<0.0001
SD	1.11	(1.07 to 1.14)	<0.0001
Normalized absolute successive variability score	1.02	(1.01 to 1.03)	<0.0001
Normalized percentage successive variability score	1.03	(1.02 to 1.04)	<0.0001
SD/initial	1.01	(1.01 to 1.01)	<0.0001
Coefficient of variation	1.01	(1.00 to 1.01)	<0.0001
Variability independent of mean	1.00	(1.00 to 1.01)	<0.0001

FBG, fasting blood glucose.

Figure 1

Graphical representation of HRs for all-cause mortality: (A) LDL-C, (B) HDL-C, (C) total cholesterol, (D) triglyceride, (E) mean HbA1c, (F) mean fasting blood glucose. HDL-C, high-density lipoprotein-cholesterol; LDL-C, low-density lipoprotein-cholesterol.

Development of a score-based predictive risk model based on Cox regression

A score-based predictive risk model for all-cause mortality was developed by incorporating significant predictors from multivariate analysis. One point was allocated for each significant predictor where the HR was less than 1.5, and 2 points for HRs between 1.5 and 2.5. Out of the eight measures of variability for HbA1c and FBG, SD had the highest HR and greatest statistical significance when adjusted to the multivariate model (FBG: HR=1.08, 95% CI 1.07 to 1.10, p<0.0001; HbA1c: HR=1.11, 95% CI 1.07 to 1.14, p<0.0001). It was therefore selected to be included in the mortality score. Altogether, the predictive risk model had a total score out of 25 (table 5). ROC analysis was performed, demonstrating the AUC of 0.729 (95% CI 0.727 to 0.731; online supplemental figure 2). Kaplan-Meier curve and the Kaplan-Meier curve stratified by male gender are shown in online supplemental figure 3 for patients with diabetes. The survival curve generated by the multivariate Cox regression model is shown in online supplemental figure 4.

Table 5

A score-based predictive risk model for all-cause mortality in type 2 diabetes mellitus

	Criteria	Score
Age	>70	1
Male	Male	1
Complete blood count
Neutrophil-lymphocyte ratio	>2.85	1
Baseline anemia	Present	2
Lipid profile
High-density lipoprotein-cholesterol (mmol/L)	<1.10 or>1.67	1
Total cholesterol (mmol/L)	<5.60 or>6.50	1
Total triglyceride (mmol/L)	>1.24	1
Comorbidity
Renal diabetic complication	Present	1
Ophthalmological diabetic complication	Present	1
Peripheral vascular disease	Present	1
Ischemic stroke	Present	1
Sudden cardiac death	Present	1
Atrial fibrillation	Present	1
Heart failure	Present	1
Intracranial hemorrhage	Present	1
Dementia	Present	2
Hypertension	Present	1
Chronic obstructive pulmonary disease	Present	1
Cancer	Present	1
Fasting blood glucose and HbA1C: baseline mean and measures of variability
Mean HbA1c (%)	<6.34 or>7.52	1
Mean FBG (mmol/L)	>6.12	1
SD: FBG	>1.63	1
SD: HbA1c	>0.79	1

Cut-off for age is rounded to the nearest whole number.

FBG, fasting blood glucose.

Results of machine/deep learning approaches for risk modeling

A RSF model was further performed to predict mortality outcome. The optimal tree number of the RSF model selected as 400 using a fivefold cross-validation approach to minimize the overall squared error rate in the testing set is shown in online supplemental figure 5. In addition, as shown in online supplemental figure 6 about the detailed main results of using the RSF model to predict the mortality outcome of patients with diabetes, the overall ensemble survivals (top left panel) are indicated by the red line and the Nelson-Aalen estimator is given by the green line. Brier score (0=perfect, 0.25>worse than guessing) stratified by ensemble mortality based on the inverse probability of censoring weight method is shown in the top right panel. We stratify the cohort into four groups of 0–25, 25–50, 50–75 and 75–100 percentile mortality (the overall, non-stratified, Brier score is shown by the red line). Continuous rank probability score given by the integrated Brier score divided by time is shown in the bottom left panel, while the illustration of mortality of each patient with diabetes versus observed time of mortality event was shown in the bottom right panel. The mortality events are shown as blue points, and we indicated censored observations using red points. Predicted OOB survivals and the cumulative hazard using the RSF model are shown in online supplemental figure 7. The predicted survival curves of patients with diabetes via the RSF model are shown in online supplemental figure 8 where blue curves correspond to censored observations while red curves represent the observations experiencing mortality events. The 10 most important predictors ranked by the RSF model are shown in online supplemental table 5.

Finally, we compared the survival analysis performance of the RSF model and DeepSurv as typical machine learning and deep learning approaches, respectively, over multivariate Cox model to predict the mortality outcome of the patients with diabetes using the fivefold cross-validation method. Sobol solver36 was used to sample each hyperparameter of DeepSurv from a predefined range and k-means cross-validation (k=3) was used to evaluate the performance of the parameter configuration settings. For k-means cross-validation, the dataset was split into k subsets with one subset used as the test set and the remaining as the training set to measure the prediction error. The role of the test and training set was switched until all subsets have been used as the test set, and a mean prediction error would be derived.37 Using the configuration with the largest validation C-index on the testing set to avoid models that overfit, we selected the best hyperparameters of the DeepSurv network which included: number of dense layers=4, learning rate=0.0003, ℓ2 regularization coefficient=3.25, dropout rate=0.36, exponential learning rate decay constant=0.0005 and momentum=0.86. In all instances, the ReLU activation function was applied.38

The comparative performance results of the different models are shown in table 6. Both RSF and DeepSurv models significantly outperform the multivariate Cox model (precision: 0.85 (95% CI 0.81 to 0.89), recall: 0.86 (0.82 to 0.89), AUC: 0.85 (0.82 to 0.91), C-index: 0.86 (0.81 to 0.90) for DeepSurv model, while precision: 0.85, recall: 0.87, AUC: 0.86, C index: 0.87 for RSF model) based on the same validation inputs of the risk predictors (P for trend <0.001). In addition, the Cox score (precision: 0.88 (0.83 to 0.92), recall: 0.87 (0.84 to 0.91), AUC: 0.86 (0.82 to 0.89), C-index: 0.87 (0.84 to 0.91)) demonstrated better performance than multivariate Cox model (precision: 0.75 (0.72 to 0.79), recall: 0.73 (0.67 to 0.77), AUC: 0.73 (0.68 to 0.76), C-index: 0.73 (0.66 to 0.77)). The advantages of machine/deep learning approaches over the Cox model arise from the fact of their strength to describe survival data with both linear and nonlinear effects from covariates. However, it should be noted that in comparison to DeepSurv, RSF allows influential predictors to be identified more easily by generating an ‘importance ranking’ of the variables with standard bootstrap theory. This enables the investigation of the predictive strength of associated risk predictors for clinicians to estimate the mortality probability just referring to the most important variables.

Table 6

Survival prediction performance comparison between Cox, RSF and DeepSurv model with fivefold cross-validation approach

Model	Precision (95% CI)	Recall (95% CI)	AUC (95% CI)	C-index (95% CI)
Multivariate Cox	0.75 (0.72 to 0.79)	0.73 (0.67 to 0.77)	0.73 (0.68 to 0.76)	0.73 (0.66 to 0.77)
Cox score	0.78 (0.74 to 0.82)	0.79 (0.74 to 0.83)	0.77 (0.73 to 0.81)	0.76 (0.72 to 0.82)
DeepSurv	0.85 (0.81 to 0.89)	0.86 (0.82 to 0.89)	0.85 (0.82 to 0.91)	0.86 (0.81 to 0.90)
RSF	0.88 (0.83 to 0.92)	0.87 (0.84 to 0.91)	0.86 (0.82 to 0.89)	0.87 (0.84 to 0.91)

AUC, area under the curve; RSF, random survival forests.

Discussion

In this study, we developed a machine learning-driven predictive risk model for type 2 diabetes mellitus using a multiparametric approach with data from different domains. Our novel findings are that (1) measures of variability of fasting glucose and HbA1c show similar predictive power for all-cause mortality, regardless of whether adjustments were made for initial values or mean values across follow-up; (2) a multiparametric predictive risk model incorporating variables from different domains, including baseline demographics, comorbidities and laboratory tests, measures of variability of HbA1c and FBG predicted all-cause mortality accurately and (3) machine learning-driven algorithms further improved the accuracy of the predictive models.

Numerous factors have been associated with premature mortality in patients with type 2 diabetes mellitus. Prior epidemiological studies have identified key risk factors including age, comorbidities, healthcare utilization patterns and laboratory findings.39 40 In our study, we also identified similar predictors that included advanced age, male gender, high neutrophil and low lymphocyte count, increased levels of urea, creatinine and potassium, as well as reduced levels of HDL-C, LDL-C, triglycerides, total cholesterol and sodium. Moreover, U-shaped relationships between LDL-C, HDL-C and total cholesterol were found in our cohort. These findings are in keeping with U-shaped relationships between cholesterol and all-cause mortality41 and for LDL-C42 in the general Korean populations. Similar relationships were found for HDL-C, where extremely high LDL-C levels were paradoxically associated with higher mortality.43 The association between all-cause mortality and elevated creatinine, urea and potassium, which are classic features of renal failure, is supported by evidence suggesting that the Asian population has a higher risk of developing diabetic nephropathy compared with Caucasians.11 It is widely accepted that current predictive models that have largely been developed using Western cohorts only provide moderate levels of accuracy and at times do not lend themselves relevant to disease management protocols that vary by country. Development of country/territory-specific risk prediction models allows for local population-based confounders and clinician management approaches to be incorporated into these models thus providing a more accurate risk prediction for the local population.

Diabetes mellitus is characterized by the presence of systemic chronic inflammation, which is accompanied by increased oxidative stress. To quantify the degree of inflammation, the NLR has been used as a surrogate measure, as it reflects the balance between proinflammatory and anti-inflammatory pathway activation. In our cohort, we found that raised NLR was associated with all-cause mortality risk. We extend previous findings of our group and other groups that increased NLR has been associated with insulin resistance in patients with newly diagnosed type 2 diabetes,44 the progression of diabetic nephropathy45 and complications in diabetes.46 Consequently, the increased oxidative stress environment in diabetes can induce adverse remodeling of the heart, which in turn increases the risk of HF, arrhythmias and cardiovascular mortality.47 48

Glycemic variability refers to the fluctuations in glucose levels and can be measured as a daily variation or variation between different clinical visits.49 50 Similarly, variability in HbA1c levels has been quantified. Both measures have been associated with a higher risk of complications and mortality in patients with diabetes mellitus in both randomized controlled trials and real-world settings.51–55 There are several methods that can be used to calculate variability, such as SD, CV and score based on the frequency exceeding a fixed percentage change in the absolute values. Prior studies have demonstrated the importance of such measures of variability in the prediction of adverse outcomes,16 17 but a systematic and direct comparison of different methodologies has not been made with regard to their predictive performance. In our study, eight different measures of variability for HbA1c and FBG were compared, all of which showed significant predictive values. Our findings illustrate that temporal variability in these laboratory tests is important, regardless of the methodology employed for its calculation. In our study, we also found that mean FBG did not predict mortality. Instead, all of the different measures of its variability were all predictive, suggesting that it is intermittent poor glucose control rather than chronic hypoglycemia that are more closely associated with all-cause mortality.

Standard survival model such as Cox proportional hazards model is a semiparametric analysis model to calculate the effects of observed patient’s covariates on the mortality risk outcome. The Cox model assumes the effect of each covariate is proportional. However, in many practical applications, the assumption is not true and risks losing decision information among the observed patient’s covariates. Furthermore, it cannot account for the presence of U-shaped relationships as only a single HR is derived for each covariate. Therefore, numerous nonlinear survival models were developed to better fit survival data with nonlinear log-risk functions (eg, time-encoded methods56) or learning the nonlinear relationship directly using machine learning and deep learning techniques (eg, feed-forward neural network risk-predicting methods57). RSF model19 that is constructed by an ensemble of binary decision trees has been identified as an alternative approach to Cox proportional hazard model in analyzing time-to-event survival data when the linear proportional hazard assumption is violated. DeepSurv58 whose multilayer perceptron architecture is deeper than Faraggi-Simon’s feed-forward model and minimizes the negative log Cox partial likelihood with a risk not necessarily linear, is capable to efficiently learn complex non-linear relationships between patient’s covariates and mortality outcome. For model selection among traditional Cox model, the Cox-based score model, RSF, and DeepSurv in risk prediction tasks, there exists a tradeoff: (1) traditional Cox models (as well as Cox based score models) provide good model interpretation ability but less accurate predictions since they sacrificed the consideration of nonlinear inter-dependent patterns among the variables; (2) machine learning or deep learning-based models significantly improves prediction performance especially when the size of instance cohort is rather large (n>1000) but some (eg, DeepSurv) may not provide good interpretations about the resulting predictions. Prediction accuracy and model interpretability are the two most important considerations for risk prediction model selection for clinical use. This study demonstrates the superiority of adopting RSF model for the risk prediction due to both its highest prediction accuracy and good model interpretability.

The findings of this study illustrate that machine/deep survival learning models can better capture the highly complex and nonlinear relationships between prognostic variables and an individual patient’s risk of mortality without prior variable selection or domain knowledge, compared with the traditional Cox analysis model. Application of machine/deep learning to survival analysis performs much better than the standard Cox model in predicting mortality risk of patients with diabetes mellitus. Additionally, machine/deep survival learning models will enable clinicians to provide personalized survival estimations based on the computed probability of mortality risk. In practice, medical researchers can use machine/deep survival learning models to improve overall survival prediction performance based on prognostic characteristics of the patients with diabetes mellitus and subsequently inform early efficient treatment options and even reduce mortality risk.

Strengths and limitations

The following strengths of our study should be noted. First, this was a territory-wide study with large patient numbers with complete and long follow-up of mortality over 10 years, owing to the linkage of the electronic health records to the death registry. Second, the availability of different data types including prior comorbidities, laboratory test results that included longitudinal data and drug details meant that we were able to build a comprehensive risk model for accurate prediction. Third, the application of the latest machine learning techniques was able to further improve the risk predictions of the models.

However, there are some limitations that should be noted. First, this was a retrospective study and therefore carries the potential bias, such as information bias, that is found in all studies of this type. Second, as with all studies using administrative databases, undercoding is a possibility. This was nevertheless mitigated by our definition of diabetes to include patients with the appropriate ICD coding and those who were on any diabetic medication or met the criteria of diabetes by either HbA1c or fasting glucose results. Patients with type 1 diabetes mellitus were not included given a different disease course and pathogenesis. Further research is needed to explore the potential for the present findings to be extrapolated onto patients with type 1 diabetes mellitus. Third, although the deep neural network survival learning approach demonstrates significant potential in providing much more accurate predictions, the model’s weak interpretability becomes the main obstacle for its real application in clinical practices. Investigations of developing interpretable deep survival learning models that provide highly accurate predictions with supportive explanations for patients with diabetes mellitus become our next research concentration.

Conclusion

A multiparametric model incorporating variables from different domains predicted all-cause mortality accurately in type 2 diabetes mellitus and a machine/deep learning-driven approach provided further improvements for risk prediction.

Table 1

Baseline characteristics for continuous variables

Characteristics	Mean	SD
Age	65.4	12.7
Follow-up duration (days)	3546	1208
Diabetes duration (years)	6.18	4.56
Liver function test
Alkaline phosphatase (U/L)	81.1	37.6
Alanine aminotransferase (U/L)	28.8	52.9
Total protein (g/L)	74.5	6.67
Albumin (g/L)	38.9	5.04
Complete blood count
Lymphocyte count (×109/L)	1.89	1.04
Neutrophil count (×109/L)	5.35	2.69
Neutrophil-lymphocyte ratio	3.72	4.37
Hemoglobin count (×109/L)	12.8	1.86
Lipid profile
HDL-C (mmol/L)	1.23	0.348
LDL-C (mmol/L)	3.09	0.941
Total cholesterol (mmol/L)	5.12	1.13
Triglyceride (mmol/L)	1.63	1.51
Renal function test
Creatinine (umol/L)	102	87.2
Potassium (mmol/L)	4.24	0.522
Sodium (mmol/L)	139	3.48
Urea (mmol/L)	6.96	4.11
Glycemic control
Fasting blood glucose	8.02	1.95
HbA1c	7.75	2.59

HDL-C, high-density lipoprotein-cholesterol; LDL-C, low-density lipoprotein-cholesterol.

Table 2

Baseline characteristics for discrete variables

Characteristics	Number	Percentage
Male	132 040	48.2
Baseline anemia	39 799	14.5
Antidiabetic agent
Biguanide	185 881	67.9
Sulphonylurea	173 525	63.4
Insulin	29 697	10.8
Meglitinide	27	0.01
Dipeptidyl peptidase-4 inhibitor	325	0.12
Thiazolidinedione	3698	1.35
Glucagon-like peptide-1 agonist	17	0.006
Acarbose	3292	1.20
Cardiovascular drugs
ACEI/ARB	121 786	44.5
Beta-adrenergic receptor blocker	92 309	33.7
Calcium channel blocker	109 225	39.9
Diuretic	52 096	19.0
Lipid-lowering agent	61 401	22.4
Comorbidities
Diabetic renal complication	3381	1.23
PVD	346	0.13
Diabetic ophthalmological complication	3543	1.29
Ischemic Stroke	8986	3.28
SCD	6420	2.34
AF	7772	2.84
HF	11 189	4.09
Intracranial hemorrhage	3264	1.19
IHD	26 423	9.65
Osteoporosis	137	0.050
Dementia	2842	1.04
Hypertension	64 246	23.5
Chronic obstructive pulmonary disease	818	0.299
Cancer	12 190	4.45

ACEI, angiotensinogen converting enzyme inhibitor; AF, atrial fibrillation; ARB, angiotensin receptor blocker; HF, heart failure; IHD, ischemic heart disease; PVD, peripheral vascular disease; SCD, sudden cardiac death.