The amputation and mortality of inpatients with diabetic foot ulceration in the COVID-19 pandemic and postpandemic era: A machine learning study.

This study aimed to explore the clinical characteristic and outcomes of inpatients with diabetic foot ulceration (DFU) in 2019 (prelockdown) and 2020 (postlockdown) due to the COVID-19 pandemic, at an emergency medical service unit. Prediction models for mortality and amputation were developed to describe the risk factors using a machine learning-based approach. Hospitalized DFU patients (N = 23) were recruited after the lockdown in 2020 and matched with corresponding inpatients (N = 23) before lockdown in 2019. Six widely used machine learning models were built and internally validated using 3-fold cross-validation to predict the risk of amputation and death in DFU inpatients under the COVID-19 pandemic. Previous DF ulcers, prehospital delay, and mortality were significantly higher in 2020 compared to 2019. Diabetic foot patients in 2020 had higher hs-CRP levels (P = .037) but lower hemoglobin levels (P = .017). The extreme gradient boosting (XGBoost) performed best in all models for predicting amputation and mortality with the highest area under the curve (0.86 and 0.94), accuracy (0.80 and 0.90), sensitivity (0.67 and 1.00), and negative predictive value (0.86 and 1.00). A long delay in admission and a higher risk of mortality was observed in patients with DFU who attended the emergency center during the COVID-19 post lockdown. The XGBoost model can provide evidence-based risk information for patients with DFU regarding their amputation and mortality. The prediction models would benefit DFU patients during the COVID-19 pandemic.

INTRODUCTION

The global pandemic of novel coronavirus disease 2019 (COVID‐19) has presented many challenges in managing inpatients with noncommunicable diseases, especially those with chronic diseases. Diabetic foot ulceration (DFU) is one of the most common and severe complications of long‐standing diabetes. More than 25% of diabetes patients develop DFU during their lifetime, with half of them becoming infected. Diabetic foot ulceration also imposes a huge economic burden on nations and their healthcare systems. Severe ulcers, neuroischemia, or infections are the most common cause of hospitalization among patients with foot problems. Early detection, diagnosis, and treatment are key to reducing DFU‐related morbidity and improving clinical outcomes. However, patients with DFU faced many challenges while accessing outpatient healthcare services due to several strict lockdowns and quarantine measures that were implemented during the COVID‐19 outbreak.

Although Rogers et al had proposed recommendations for managing diabetic wounds during the pandemic, many gaps remain for hospitalized patients. According to Caruso et al, there was a significantly higher proportion of patients with DFU emergency admissions during the 2020 lockdown compared to 2019, and there was a higher risk of amputation. Liu et al showed that the COVID‐19 epidemic had serious effects on DFU care; the time that patients reported the onset of DFU during their medical visits increased significantly, and the number of inpatients decreased significantly. In their study, however, there were no significant differences in mortality and amputation rates among inpatients. Shin et al also reported that whether in Manchester, United Kingdom, or Los Angeles, United States, the number of inpatients affected by the COVID‐19 outbreak has dropped significantly, and specialists are taking steps to prevent necessary hospitalizations. Further, patients with DFU who get an accidental wound are less likely to receive emergency care. Therefore, there is a lack of substantial clinical evidence on whether DFU inpatients have worse clinical outcomes during the COVID‐19 pandemic, especially as the loss of life and limbs among patients hospitalized with foot problems in urban designated emergency medical centers has not been reported.

Despite being a once‐in‐a‐century health crisis, the effects of COVID‐19 could last several years or even decades. Therefore, for DFU patients and multiple stakeholders with an interest in wound care, these difficulties may be faced for a long time, and new strategies should be created. One hypothesis that cannot be neglected is that the risk factors of inpatients with foot ulcers may have changed or even have unique new risk factors during the pandemic. Therefore, the awareness of the early warning signs of DFU risk in the new period will help in the prevention and care of DFU. Although statistical models have been extensively used in the study of chronic diseases such as multiple linear regression and logistic regression analyses, the algorithm is relatively simple, and the results cannot accurately give specific probability values. In recent years, with the development of artificial intelligence (AI), machine learning algorithms have displayed their advantages in predictions and recognition of the risk for chronic diseases. , , , It is possible to build predictive models of the related diseases by relying on electronic health record (EHR) data. Unfortunately, there are no reports of machine learning models for predicting amputation and mortality in DFU inpatients.

Therefore, it is necessary to establish a machine learning model that can be used to predict the clinical outcomes of hospitalized patients with a diabetic foot ulcers. The model will provide early warning of risk factors for amputation and death of DFU inpatients at the time of admission, offer new ideas for further understanding the new risks of diabetic ulcer patients in the postepidemic era, and supply appropriate recommendations for the prevention and care of DFU. Hence this study compared and analyzed EHR over the same period of 2020 (postlockdown) and 2019 (prelockdown) at an emergency medical service unit. It explored the clinical characteristics and outcomes of inpatients with DFUs under the COVID‐19 outbreak. The study addressed the commonalities associated with the different clinical outcomes in DFU inpatients and established a new machine‐learning‐based predictive model for clinical outcomes in diabetic foot inpatients.

RESEARCH DESIGN AND METHODS

Electronic health records data were collected from a city‐designated emergency tertiary medical center in a metropolis with a resident population of more than 12 million. This retrospective study analyzed consecutive patients with typical DFUs who met the International Working Group on the Diabetic Foot (IWGDF) guidelines diagnostic criteria during the postlockdown period from January 2020 to May 2020 and the prelockdown period from January 2019 to June 2019. All presenting patients meeting the following criteria were included: WIFi wound classification, grade 1‐3, class 1‐3 ischemia, and class1‐3 ft infection requiring emergency admission; Ultrasound CT angiography or magnetic resonance angiography (MRA) were applied to confirm the presence of lower limb artery lesions. Finally, 23 patients were included in 2019 (prelockdown) and the same number in 2020 (postlockdown). All patients enrolled in this study were received the current DFU standard of care during hospitalization involved as following principles: (a) pressure relief, (b) debridement, (c) infection management, and (d) revascularization when indicated.

The data was preprocessed from the EHR. Finally, 31 candidate indicators of interest were prepared based on clinical knowledge and previous studies. These variables include patient demographics (age, and sex, etc), laboratory test results (blood glucose, and white blood cells, etc), diabetic complications or comorbidities, clinical outcomes (amputation and mortality), and prehospital delays. Statistical methods were used to analyze the differences of various indicators in different periods (prelockdown vs post lockdown). These indicators were used as predictors to build machine learning models to compare the performance of different machine learning algorithms in predicting the risk of amputation and death in hospitalized diabetic foot patients. This work was approved by the Ethical Committee Board of Chongqing University Central Hospital, China.

STATISTICS ANALYSIS

Descriptive analysis was used to summarize demographic and clinical characteristics. Continuous variables that conformed to a normal distribution were expressed as mean (standard deviation), and differences between groups were analyzed using t‐test or one‐way ANOVA. Data not conforming to the normal distribution was presented as median (interquartile range, IQR), and were log‐transformed when necessary to meet normality. The F test was performed on the transformed data, and nonparametric tests were used if normality was not attained. Categorical variables were presented as a No. (%), and the Chi‐squared or Fisher's exact tests were employed to compare differences in distribution. The primary outcomes of interest in this study were prehospital delay, in‐hospital amputation rate, and mortality. Hence, survival curves with Kaplan–Meier survival analyses were calculated and plotted for comparison of the clinical outcomes. P˂0.05 was considered statistically significant. The impact of demographics and clinical characteristics on hospital outcomes was assessed using a machine learning model. The accuracy of the model was assessed by calculating the area under the receiver operating characteristic (AUROC) curve.

PREDICTION MODELS

In this study, six widely used machine learning‐based algorithms were available as a basis for developing prediction models. The accuracy of logistic regression (LR), support vector machine (SVM), random forest (RF), gradient boosting decision tree (GBDT), artificial neural network (ANN), and extreme gradient boosting algorithm (XGBoost) models were established and compared. These algorithms have been previously demonstrated to be robust and applicable to big data sets. A total of 31 different variables were used as inputs into the prediction model. Finally, the most accurate model was used to analyze the gradient of risk factors for amputation and mortality of diabetic foot inpatients during the COVID‐19 pandemic.

MODEL VALIDATION AND COMPARISON

In the modeling phase, 13 samples (10 subjects without amputation and 3 subjects with amputation) of the 2020 dataset were randomly picked to build the training set to train the prediction model for amputation. The remaining 10 samples (seven subjects without amputation and three subjects with amputation) were selected as the test set. A total of 13 samples (11 survivals and 2 mortalities) were selected as the training set, and the remaining 10 samples (eight survivals and two mortalities) were selected to set up the test group for the prediction models of mortality. All six models were trained and validated in the above training set and test set based on all variables in Table 1.

TABLE 1

The clinical characteristics of inpatients

Risk factors	2019	2020	P
	Prelockdown	Postlockdown
Clinical characteristics
Age (years)	66.26 ± 10.55	67.00 ± 9.80	.807
Sex (female/male)	6/17	5/18	.730
Diabetic duration (years)	11.57 ± 7.83	10.94 ± 9.19	.809
Prehospital delay, median (IQR), (days)	5 (0–14)	20 (2–70)	.022*
Charlson, median (IQR)	7 (4–9)	7 (4–8)	.079
Previous DFU (yes/no)	3/20	9/14	.043*
Hypertension (yes/no)	15/8	17/6	.522
Cardiac heart disease (yes/no)	15/8	12/11	.369
Cerebral infarction (yes/no)	5/18	6/17	.730
Diabetic neuropathy (yes/no)	22/1	17/6	.096
Diabetic retinopathy (yes/no)	11/12	6/17	.127
Diabetic nephropathy (yes/no)	13/10	12/11	.767
Heart failure (yes/no)	11/12	8/15	.369
Foot Gangrene (yes/no)	13/10	13/10	1.000
Infection in others organs (yes/no)	11/12	7/16	.227
Smoking habit (yes/no)	13/10	14/9	.765
Alcohol misuse (yes/no)	7/16	6/17	.743
Laboratory test
HbA1c (%)	9.70 ± 3.10	9.40 ± 3.50	.632
HbA1c (mmol/mol)	83 ± 10	79 ± 15	.632
Haemoglobin (g/l)	126.09 ± 22.94	109.04 ± 23.74	.017*
WBC, median (IQR), (103/μL)	8.29 (6.47–11.19)	10.2 (6.89–15.94)	.123
Serum albumin (g/l)	37.25 ± 7. 56	34.95 ± 6.53	.286
Total cholesterol (mmol/l)	4.17 ± 0.77	4.02 ± 1.30	.649
HDL (mmol/l)	1.09 ± 0.29	0.89 ± 0.34	.068
LDL cholesterol (mmol/l)	2.09 ± 0.68	2.24 ± 0.97	.576
Triglycerides (mmol/l)	1.29 ± 0.47	1.66 ± 0.82	.087
hs‐CRP, median (IQR), (mg/L)	17.90 (2.30–48.00)	67.45 (13.60–137.20)	.037*
WIfI classfication
Wound (3/<3)	7/16	3/20	.284
Ischemia (3/<3)	2/21	1/22	1.000
Foot Infection (3/<3)	1/22	7/16	.047*

Note: P < .05 was considered statistically significant (*P < .05).

Abbreviations: DFU, diabetic foot ulcer; Foot Infection = 3, foot infection with systemic inflammatory response syndrome (SIRS); HbA1c, glycosylated hemoglobin; HDL, high‐density lipoprotein; hs‐CRP, high‐sensitivity C‐reactive protein; LDL, low‐density lipoprotein; WBC, white blood cells.

In the model training phase, with the purpose of fine‐tuning hyperparameters of the models and avoiding overfitting, 3‐fold cross‐validation and grid search were performed within the training set, where the training set was randomly divided into three subsets; for each iteration, one subset was selected as the testing set, and the rest were selected as the training set. The hyper‐parameter set of each model was selected with the best validation accuracy according to the mean value of the 3‐fold cross‐validation. Performance metrics were calculated to validate the model's predictive power and to compare the performance of different models. The performance of the prediction model was assessed using the Area under Curve (AUC, 0.5 = chance and 1 = perfect discrimination), which illustrates the trade‐off between the true positive rate (TPR) and false‐positive rate (FPR) specificity at a variety of threshold settings. Nevertheless, AUC assumes that the prediction of positive and negative samples is of equal importance. Other evaluation parameters were used to evaluate the performance of the prediction models, including accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). The following formula was used to calculate the evaluation index: where TP and FP are the numbers of true and false positives; TN and FN are the true and false negatives.

VARIABLE RANKING

The study aimed to find out the contribution of each variable to the predicted results based on XGBoost (the best predictive model among the six tested models), which is an efficient and flexible machine learning algorithm that solves scientific problems fast and accurately. The use of XGBoost increased in popularity among scientific research and has been successfully applied to the prediction of diseases (such as kidney disease and HIV‐1 tropism) due to its excellent performance characteristic. The high performance of XGBoost shows that it can be effectively used to analyze the influencing factors. XGBoost prediction model generates leaf nodes that provide information for decision‐making based on the rules and thresholds and calculates the information gained in the process of generating the leaf nodes, which was used to assess the relative importance of influencing factors affecting the outcomes among inpatients with DF. The information gained reflects the degree to which variables reduce the uncertainty of training set classification; the higher the value, the more critical to the prediction results. This was calculated using the following equation: where and represent the sum of the first derivatives of samples contained in leaf nodes, and represent the sum of the second derivatives of samples contained in leaf nodes, whereas and are parameters used to control the complexity of the mode.

PATIENT AND PUBLIC INVOLVEMENT

Patients or the public were not involved in the design, or conduct, or reporting, or dissemination plans of our research.

RESULTS

Six of the 23 patients admitted prelockdown were female, with a mean age of 66 years and a diabetes duration of 11.57 years, and five of the 23 patients admitted after postlockdown were female, with a mean age of 67 years and a diabetes duration of 10.94 years. In addition, there was no significant difference in other clinical characteristics, including the type of complications, except for the time of prehospital delay in patients admitted before and after lockdown. As shown in Figure 1, the length of delay in hospitalization increased significantly after the lockdown in 2020 (10 days vs 5 days, P = .022). There was a significant increase in inpatient mortality in 2020 (17.4% vs 0%, P = .024), and although the rate of minor‐amputations displayed an increasing trend, there was no statistically significant difference. There was a significant increase in the proportion of inpatients with a history of DFU (P = .043), haemoglobin concentration (P = .017), and high‐sensitivity C‐reactive protein (hs‐CRP) levels (P = .037) after the lockdown in 2020 (Table 1). Besides, there were no significant differences in the wound size and ischemia, but an increasing trend existed in systemic inflammatory response syndrome (SIRS) in 2020 (30.4% vs 4.3%, P = .047).

FIGURE 1

The prehospital delay and clinical outcomes of inpatients (A) The prehospital and mortality were significantly increased in 2020 (P < .05), but no significant difference in amputation. (B) Kaplan–Meier survival analysis shows a significant difference in mortality (P = .024)

The prediction results and risk factor score

All the values of the performance indexes of the six models in the test set are presented in Table 2. The AUC varied between 0.60 and 0.86 for different amputation prediction models, whereas for predicting mortality, the AUC varied from 0.56 to 0.94. The results show that the prediction performance was better when predicting the risk of mortality compared to amputation.

TABLE 2

The performance indexes of the six models in the test set

	LR	SVM	RF	GBDT	XGBoost	ANN
Model evaluation criteria for amputation
AUC	0.76	0.60	0.67	0.67	0.86	0.71
Accuracy	0.70	0.70	0.80	0.70	0.80	0.70
Sensitivity	0.33	0.00	0.67	0.33	0.67	0.33
Specificity	0.86	1.00	0.86	0.86	0.86	0.86
PPV	0.50	0.00	0.67	0.50	0.67	0.50
NPV	0.75	0.70	0.86	0.75	0.86	0.75
Model evaluation criteria for mortality
AUC	0.81	0.56	0.88	0.88	0.94	0.69
Accuracy	0.70	0.80	0.80	0.90	0.90	0.70
Sensitivity	0.50	0.00	0.50	1.00	1.00	0.50
Specificity	0.75	1.00	0.88	0.88	0.88	0.75
PPV	0.33	0.00	0.50	0.67	0.67	0.33
NPV	0.86	0.80	0.88	1.00	1.00	0.86

Abbreviations: ANN, artificial neural network; AUC, area under curve; GBDT, gradient boosting decision tree; LR, logistic regression; NPV: negative predictive value; PPV, positive predictive value; RF, random forest; SVM, support vector machine; XGBoost, extreme gradient boosting algorithm.

The results demonstrated that the XGBoost performed best in all models in predicting amputation and mortality with the highest AUC (0.86 and 0.94), accuracy (0.80 and 0.90), sensitivity (0.67 and 1.00), and NPV (0.86 and 1.00) (Table 2). On the contrary, SVM was the worst at predicting amputation and mortality that can even be considered to have no predictive power. The remaining four models did not perform, as well as extreme gradient boosting algorithm in predicting the chances of amputations but were acceptable. However, the GBDT and RF performed well in predicting mortality with an AUC of 0.88 and 0.94, respectively. By contrast, LR and ANN were relatively mediocre, with an AUC of 0.81 and 0.69.

The relative rankings of importance of each baseline variable in the amputation and mortality prediction model based on XGBoost were obtained according to gain information (Figure 2). The amputation prediction models based on gain information in 2019 and 2020 show that the three most important baseline variables in 2019 were white blood cells, blood potassium, and prehospital delay, while in 2020, they are prehospital delay, ischemia, and serum albumin. The three most important baseline variables in the mortality prediction model were other infections, age, and foot infection.

FIGURE 2

The distribution of feature importance scores under the extreme gradient boosting (XGBoost) model (A, B) The top 10 most important features indicated by XGBoost on the 2019 and 2020 amputation, respectively. (C) The top 10 most important features indicated by XGBoost on the 2020 mortality

DISCUSSION

This study reports the clinical characteristics and outcomes of hospitalized DF patients over the same period before and after the COVID‐19 outbreak. The results show that DFU inpatients in urban emergency units had a significantly increased delay of hospitalization during the COVID‐19 lockdown. The reasons for these reflect a strict blocking and insulated management, shortage of medical workforce, and patients scared by COVID‐19 with poor health‐seeking behaviour. Hospitalization delays exist in all aspects of the management pathway under COVID‐19, a sharp decrease in admission as patients avoided exposing themselves to infection. General physicians and nurses lack of specialized knowledge, even in top of DFU care clinics, was one of the important factors for delay of admission. This undesired clinical outcome may be due to a significantly prolonged delay in visiting the hospital due to the epidemic. The previous report has supported this view. Yan et al's study showed that prehospital delay increased the risk of death in patients with DFU. Therefore, novel strategies and referral recommendations should be explored during COVID‐19. The adoption of standardized limits for referral and treatment times, exploring missed opportunities for diagnosis, and investigations of novel strategies for providing specialist care are necessary to reduce delays. Another interesting discovery was that patients who had a history of DFU had higher rates of hospitalization and faster visits during the lockdown. Patients with an experience of DFU had good health‐seeking behaviorr to avoid worsening. These patients also exhibited better clinical outcomes. Although there was an increasing trend in minor amputation, the difference was not significant. There was 17.4% mortality in the emergency service unit, which was lower than in patients with a diabetic foot who underwent emergency hospitalization (23.5%). It could be different based on the experience of lower limb salvage in the DFU care center. After lockdown, the DFU patients had a poor nutrition state and low hemoglobin. Zhang et al showed that poor nutrition is detrimental to the prognosis of DFU. The foot infection with systemic inflammatory response syndrome (SIRS) was more severe in 2020, according to the WIfI classification. Serum CRP level also was associated with the status of DFU patients postlockdown. Patients presenting with SIRS had a poor prognosis, including higher in‐hospital mortality and minor amputations. Clinicians should be aware of the clinical factors that can develop and those that affect the prognosis in treating patients with limb‐threatening foot infections. The COVID‐19 pandemic has caused a deleterious effect on DFU patients resulting in more severe infections and foot emergencies.

The machine learning model has been widely used for disease model establishment and prognosis prediction. Yuan et al established a risk prediction model for heart failure based on the stochastic forest algorithm. Ayer et al established a logistic regression model and an artificial neural network (ANN) model to evaluate the risk of breast cancer and compared the characteristics, advantages, and limitations of the two models. Ruan et al introduced XGBoost to build a machine learning model for predicting the risk of hypoglycemia in diabetic inpatients, with a model evaluation precision of 0.88. In the present study, XGBoost performed best among all models in predicting amputation and mortality with the highest AUC, accuracy, sensitivity, and NPV. Therefore, XGBoost was used to evaluate the importance of the risk factors. The three most important baseline variables in the amputation prediction model were white blood cells, blood potassium, and prehospital in prelockdown, whereas prehospital, foot ischemia, and serum albumin were the factors in postlockdown. Furthermore, the three most important baseline variables in the mortality prediction model were nonfoot infection, age, and foot infection. These show that DFU patients with either nonfoot or foot infections require emergency treatment. Patients who require emergency treatment often have several complications and comorbidities, such as nonfoot infections, heart disease, and kidney disease. Therefore, single knowledge of podiatric or internal medicine is not enough, and multidisciplinary participation is required. The important role of local or systemic infection in DFU for mortality has also been identified. ,

It is challenging to identify the biggest threat to a patient's life simply by relying on a nonspecialist of DFU. However, it is difficult to achieve multidisciplinary consultation due to the shortage of medical resources during the COVID‐19 epidemic. The inpatient mortality and the risk of amputation can easily be analyzed, based on the machine learning model and first aid done. This may provide a new avenue to improve clinical outcomes in patients with acute DFU, as we can rely on machine learning outcomes to prioritize the management of high‐risk factors.

This study indicates that the XGBoost model still exhibits stable and excellent performance, despite the small amount of data used for training. Regarding the predictive results of amputation and mortality, XGBoost scored the highest on four of the six evaluation indexes on both counts, which means that the XGBoost model is a suitable tool in predicting the risk of death and amputation in high‐risk diabetic foot patients during the COVID‐19 pandemic. Nevertheless, certain limitations of the model exist; despite 3‐fold cross‐validation, there is still a lack of an independent external validation cohort. In addition, the relative importance of variables based on XGBoost is not equal to that in the causal chain; prospective studies are required to validate the role of these model‐based risk factors in prognosis. Despite these limitations, the developed model is expected to be used to screen high‐risk patients in advance and provide guidance for follow‐up treatment, and the data required for the model can be obtained when the patient is admitted to the hospital and receives an initial routine examination.

CONCLUSION

A longer delay in admission and a higher risk of mortality was observed in patients with DFU who attended the emergency center during the COVID‐19 post lockdown. Timely referral and proper management, especially for poor nutrition and systemic or foot infection, could prevent the worsening of DFU. The XGBoost model can provide evidence‐based risk information for patients with DFU regarding their amputation and mortality. The prediction models would be beneficial for DFU during the COVID‐19 pandemic.

CONFLICT OF INTEREST

No potential conflicts of interest relevant to this article were reported.

ETHICS STATEMENT

The study was approved by the Committee for Research Ethics of Chongqing University Central Hospital.

CONSENT FOR PUBLICATION

This manuscript has been read and its submission approved by all the coauthors.