Prediction model based on machine learning for short- and long-term adverse events in left atrial appendage closure.

Background: At present, the prediction of adverse events (AE) had practical significance in clinic and the accuracy of AE prediction model after left atrial appendage closure (LAAC) needed to be improved. To identify a good prediction model based on machine learning for short- and long-term AE after LAAC.
Methods: In this study, 869 patients were included from the Department of Cardiovascular Medicine of Shanghai University of Medicine and Health Sciences Affiliated Zhoupu Hospital during 2017 and 2021. Univariate and multivariate analyses were conducted for short-term AE after LAAC to determine possible risk factors related with AE. We compared 8 machine learning algorithms for prediction short-term AE, and XGBoost was found to have the best performance. In addition, Cox-regression was used for long-term AE to find out the risk factors and establish a prediction model.
Results: In univariate and multivariate analysis, body mass index (BMI) [odds ratio (OR) =0.91], congestive heart failure, hypertension, age ≥75 years, diabetes, stroke2 attack (CHADS2) score (OR =0.49) and bleeding history or predisposition, labile international normalized ratio (INR), elderly, drug/alcohol usage (BLED) score (OR =1.71) were shown to be significant risk factors for short-term AE. The XGbosst algorithm was used to predict short-term AE based on 15 possible risk factors. For long-term AE, Cox regression was used for the prediction. The CHADS2 score [hazard ratio (HR) =1.43], hypertension (HR =2.18), age more than 75 (HR =0.49), diabetes (HR =0.57), BLED score (HR=0.28), stroke (HR =19.8), hepatopathy (HR =3.97), nephropathy (HR =2.93), INR instability (HR =4.18), drinking (HR =2.67), and drugs (HR =2.36) were significant risk factors for long-term AE. The XGBoost had a good receiver operating characteristic (ROC) curve and area under the curve (AUC) was 0.85. The accuracy of the XGBoost model stayed at nearly 0.95. Conclusions: In short- and long-term AE, CHADS2 score and BLED score were the most obvious risk factors. Several other risk factors also played roles in AE of LAAC. The incidence of long-term AE is under 15% and LAAC is effective and safe. The XGBoost model had good prediction accuracy and ROC curve. 2022 Journal of Thoracic Disease. All rights reserved.

Introduction

Atrial fibrillation (AF) is the most common arrhythmia worldwide, affecting 32 million people globally and 5 million in the United States. It is estimated that the annual medical cost of AF in the United States is $26 billion (1). In a study of AF from China in 2004, the number of patients with AF accounted for 0.77% of the total survey population; the number of patients was about 10 million, and AF was significantly higher in men than in women (2). The disease has significant effects on quality of life, and is associated with heart failure, thromboembolism, stroke risk, and all-cause mortality (3). The proportion of stroke caused by AF suddenly increased from 1.5% in the age of 50–59 years to 23.5% in the age of 80–89 years. A large number of studies have shown that 91–100% of patients with non-valvular AF have thrombosis in the left atrial appendage (LAA) (4,5). The treatment of AF has become a real challenge for cardiologists and electrophysiologists. Although oral anticoagulants are currently the standard protocol for preventing AF from stroke, due to various reasons such as bleeding risk, compliance, and cost, 50% of AF patients remain unsuitable for oral anticoagulant therapy (5). For AF patients who are unable to tolerate oral anticoagulants, left atrial appendage closure (LAAC) is an alternative therapy to lower the risk of adverse events. The 2019 American Heart Association (AHA) guidelines for the management of patients with AF and the 2020 European Society of Cardiology (ESC) guidelines for the diagnosis and management of AF clearly stipulate that LAAC is suitable for patients with non-valvular AF at high risk of stroke (6).

The LAA is the most common source of thrombosis in patients with post-stroke AF. The special morphology of left atrium, trabecular meshwork, pectoral muscle, atrial related inflammation, atrial remodeling, and hypercoagulable state are related to thrombosis (7). At present, oral anticoagulants (OAC) such as warfarin or direct oral anticoagulants (DOACs) are used for the preventative treatment of AF and patients with a congestive heart failure, 75 years, hypertension, diabetes, vascular disease, transient ischemic attack (TIA), age 65 to 74 years, women (CHA2DS2) VASc score over 1 point (8). The LAAC is an effective and safe method to prevent stroke and systemic thrombosis caused by AF. In addition to preventing stroke, studies have suggested that epicardial LAA occlusion can reduce the burden of arrhythmia and have a significant impact on atrial remodeling (9,10).

In LAAC, the probability of air embolism is 5%, but in most clinical cases, this is a hidden event. Air embolism may lead to some mild and serious events, such as transient coronary ischemia, hypotension, stroke, and even death. Pericardial effusion is the most common complication of LAAC (10). When the exudation volume is large, it can lead to cardiac tamponade. Pericardiocentesis and drainage is an urgent problem to be solved; where necessary, pericardial decompression or thoracotomy should be promptly performed. Residual shunting of the occluder [PEO device leakage (PDL)], occluder surface, and intraoperative thrombosis of the left atrium are also typical complications of LAAC (11).

It was reported that device-related thrombus (DRT) after LAAC was associated with ischemic events. Patient- and procedure-specific factors were associated with the risk of DRT (10). There were several previous researches on the risk factors of AE after LAAC, however, limited research explored the prediction of AE after LAAC. The prediction was necessary which might help to prevent AE in practice.

In recent years, machine learning became an important method of model prediction. At the same time, with the software development, machine learning contained more and more methods. Therefore, when choosing the machine learning model, it was necessary to compare the accuracy with the receiver operating characteristic (ROC) curve of the models.

In view of the impact of LAAC complications on the clinical effect, we attempted to use machine learning and survival analysis to explore the influencing factors of short- and long-term complications, respectively, to put forward targeted preventive measures. We also conducted prediction according to the existing data, which can help the patients receiving LAAC to prevent complications. We present the following article in accordance with the STARD reporting checklist (available at https://jtd.amegroups.com/article/view/10.21037/jtd-22-499/rc).

Methods

Research design

This research is a retrospective cohort study. We selected eligible patients as samples. The inclusion criteria were: (I) patients aged more than 40 years old; (II) patients received LAAC; (III) patients accepted telephone follow-up; (IV) patients with CHADS2 score ≥2. Most of included data was recorded during hospitalization and supplementary information was added through telephone follow-up. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by institutional ethics board of Zhoupu Hospital, Pudong New Area, Shanghai (No. 2021-C-135-E01). Informed consent was taken from all the patients.

Dataset

The data set of this study was taken from the Department of Cardiovascular Medicine of Shanghai University of Medicine and Health Sciences Affiliated Zhoupu Hospital. Patients with left atrial occlusion treated by LAAC from 2017 to 2021 were collected.

In order to evaluate the factors that affect the efficacy and prognosis of LAAC, the data set included 15 factors that affect the complications and complications of LAAC patients: CHADS2 score, history of myocardial infarction, hypertension, age over 75 years, history of diabetes, history of cerebral infarction, chronic heart disease, bleeding score, stroke history, liver injury, kidney disease and bleeding history, international normalized ratio (INR) instability, and history of alcohol and drug use. Finally, 869 patients were included in this study.

Process of LAAC surgery

The LAAC procedures were performed under angiographic and echocardiographic guidance and according to the instructions for use (IFU). All patients underwent surgery under general anesthesia. Atrial septal puncture was performed under the guidance of transesophageal echocardiography (TEE), and heparin was given after surgical success to maintain an activated clotting time (act) of >250 s. A 5F pigtail catheter was used for LAA angiography [right anterior oblique (RAO) 30° + head or foot 20°] to reveal the shape, size, and adjacent relationship of the LAA. The LAA depth and mouth diameter were measured by TEE at 4 angles of 0°, 45°, 90°, and 135°. The LAA occluder was selected according to the LAA depth and mouth diameter measured by angiography and TEE. The diameter of the occluder was set at 8–30% larger than that of the LAA. Successful occlusion was defined as LAA occlusion at all auricular lobules or only a small residual shunt (diameter ≤5 mm). A traction test was performed to verify that the occluder was in a stable position; the compression ratio of occluder diameter was greater than 8%. The computer-aided design evaluation of LAAC surgery is shown in and the TEE with angiography before and after surgery are shown in .

Figure 1

Computer-aided design evaluation of post-LAAC. LA, left atrium; LAAC, left atrial appendage closure.

Figure 2

Transesophageal echocardiography and angiography before and after LAAC. (A) Baseline TEE; (B) LAAC TEE; (C) baseline angiography; (D) LAAC angiography. LAAC, left atrial appendage closure; TEE, transesophageal echocardiography.

Evaluation of adverse events after LAAC surgery and in follow-up

The complications after LAAC were observed by esophageal ultrasound. The complications included surface endothelialization, posterior shoulder leakage, lower edge shunt, suspected pulmonary vein thrombosis, suspected thrombus on the surface of the occluder, pericardial effusion, patent foramen ovale, mitral regurgitation, and left atrial stasis (12).

The first follow up time was between 15 and 498 days after surgery. Long-term adverse events (AE) included pericardial effusion, heart failure, stroke, aneurysm, death, pneumothorax, valvular heart disease, heart rupture, occlusion, and displacement. Since the variety of AE, there prediction should be based on multi classification prediction.

Statistical analysis

We compared 8 different machine learning algorithms, including extreme gradient boosting (XGBoost), decision tree (DT), support vector machine (SVM), multiple adaptive regression splines (MARS), artificial neural network (ANN), boosting tree (BT), random forest (RF), and k-nearest neighbor (KNN). In the comparison of 8 algorithms, XGBoost was revealed as the best choice.

The XGboost is an optimized distributed gradient enhancement library designed to be efficient, flexible, and portable. The machine learning algorithm is implemented in the gradient boosting framework. The XGboost provides a parallel tree push (also known as gbdt, GBM), which can quickly and accurately solve many data science problems. The same code runs on major distributed environments (Hadoop, SGE, MPI) and can solve the problems in billions of examples.

The theory of the XGBoost algorithm could be summarized as follows: Assume a training dataset D={(xi, yi), i=1..n} of the size n, where xi=(xi1 , xi2 ,…, xℑ) denoted an m-dimensional feature vector with the corresponding (output) category yi:

where K represented the number of trees, f (x) represented the score which was associated with the model’s k-th tree, and F denoted the space of scoring functions available for all boosting trees.

To compare the performance of different machine learning algorithms, accuracy and ROC curve with AUC value were needed.

XGBoost model fitting and validation

Firstly, we divided the original data into training data and test data at a ratio of 8:2. The training data was used to train the XGBoost model, and the test data was used to verify the model.

We validated all algorithms through 10-fold cross validation. In addition, 15 sub-data sets are obtained through 10-fold cross validation, and the performance of deep learning and traditional machine learning was compared.

To select features and evaluate model fitting, variable importance diagrams and ROC curves were created. In addition, the prediction of the XGBoost model was evaluated by confusion matrix, and the sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), kappa value, and accuracy were assessed.

In this research, R language (The R Foundation for Statistical Computing, Vienna, Austria) was used for the algorithm implementation. P<0.05 was considered statistically significant.

Results

According to AE, we listed the demographic data in . Male participants were more represented than females in both AE groups and had no significant difference (χ2=0.35, P>0.05). The average age was similar in 2 groups and esophageal ultrasound results had no difference (F=0.224, P>0.05). The body mass index (BMI) value in 2 groups had difference (χ2=17.889, P<0.05) while esophageal ultrasound had no difference (χ2=1.0731, P>0.05).

Table 1

Demographics and risk factors of included samples received LAAC surgery during 2016–2021

Items	Adverse events		χ2/F
	Yes	No
Demographic
Gender			χ2=0.35
Male	20	496
Female	11	342
Age, years	74.9±9.09	74.12±9.13	F=0.224
BMI, kg/m2			χ2=17.889*
<18.5 (underweight)	13	124
18.5–24.9 (normal)	10	354
25–29.9 (overweight)	3	230
≥30 (obese)	6	129
Esophageal ultrasound			χ2=1.0731
No abnormality	28	698
Abnormality	3	140
Risk factors
CHADS2 score			χ2=11.3186642
1	0	9
2	1	38
3	4	80
4	5	140
5	10	166
6	0	167
7	6	139
8	5	81
9	0	18
MI			χ2=0.44983
Yes	20	490
No	11	348
Hyt			χ2=0.080142
Yes	25	658
No	6	180
Age75			χ2=0.12003
Yes	14	405
No	17	433
DM			χ2=0.044022
Yes	7	203
No	24	635
CI			χ2=1.654
Yes	19	415
No	12	423
CVD			χ2=0.18003
Yes	25	700
No	6	138
BLED score			χ2=5.232
0	0	22
1	1	95
2	7	205
3	13	281
4	8	195
5	1	33
6	1	7
Stroke			χ2=2.1593
Yes	19	401
No	12	437
Hep			χ2=0.025743
Yes	1	23
No	30	815
Ked			χ2=0.010886
Yes	1	30
No	30	808
Blood			χ2=2.0396
Yes	1	96
No	30	742
INR			χ2=0.07146
Yes	2	65
No	29	773
Alc			χ2=0.75541
Yes	5	93
No	26	745
Drug			χ2=0.052543
Yes	17	442
No	14	396

*P<0.05. LAAC, left atrial appendage closure; BMI, body mass index; CHADS2, congestive heart failure, hypertension, 75 years, diabetes, stroke/TIA, vascular disease, age 65 to 74 years, gender category (women); MI, history of myocardial infarction; Hyt, hypertension; age75, age over 75 years; DM, history of diabetes; CI, history of cerebral infarction; CVD, chronic heart disease; BLED score, bleeding score; Hep, liver injury; Ked, kidney disease; Blood, bleeding history; INR, international normalized ratio instability; Alc, drinking history.

In addition, we compared the 15 included risk factors and the results showed that all 15 risk factors had no difference between 2 AE status.

As shown in , 55.58% patients received a WATCHMAN device (Boston Scientific Corp., Marlborough, MA, USA) and most patients received general anesthesia (81%). Patients without AE had the device size 26 with range 24–30 mm and AE group had the device size 27 with range 24–30 mm. The no AE group’s fluoroscopy time was 9.1 (range, 6.2–12.05) min and the AE group’s was 8 (range, 7.2–9.23) min. The no AE group’s X ray-dose was 227.5 (range, 100.5–424.45) and that of the AE group was 287.6 (range, 18.05–443.3).

Table 2

Cox regression of complication in the initial follow-up and risk factors

Variables	Coefficient	HR	SE of HR	Z	P value
Model fit
Gender	−0.115	0.892	0.22	−0.521	0.602444
Weight	0.0006	1.01	0.0004	1.459	0.144436
Height	0.0003	1.00	0.001	0.362	0.717353
CHADS2	0.358	1.43	0.134	2.684	0.007279*
MI	−0.142	0.87	0.22	−0.644	0.51986
Hyt	0.779	2.18	0.259	3.009	0.002618*
Age75	−0.711	0.49	0.315	−2.256	0.024064*
DM	−0.561	0.57	0.273	−2.058	0.039596*
CI	−1.30	0.272	1.02	−1.281	0.200046
CVD	−0.088	0.916	0.255	−0.344	0.730814
BLED	−1.27	0.28	0.112	−11.347	<0.000002*
Stroke	7.58	19.8	1.48	5.121	0.000000304*
Hep	1.38	3.97	0.387	3.562	0.000368*
Ked	1.07	2.93	0.37	2.903	0.003695*
Blood	0.382	1.47	0.381	1.003	0.316057
INR	1.43	4.18	0.358	3.997	0.0000641*
Alc	0.981	2.67	0.259	3.783	0.000155*
Drug	0.859	2.36	0.224	3.84	0.000123*
Model evaluation
Likelihood ratio test	168.8 with P<0.05
Wald test	185.8 with P<0.05
Logrank test	264.3 with P<0.05

*P<0.05. HR, hazard ratio; SE, standard error; CHADS2, congestive heart failure, hypertension, 75 years, diabetes, stroke/TIA, vascular disease, age 65 to 74 years, gender category (women); MI, history of myocardial infarction; Hyt, hypertension; age75, age over 75 years; DM, history of diabetes; CI, history of cerebral infarction; CVD, chronic heart disease; BLED score, bleeding score; Hep, liver injury; Ked, kidney disease; Blood, bleeding history; INR, international normalized ratio instability; Alc, drinking history.

Based on the AE after LAAC surgery, we implemented 15 variables related with complications after LAAC in a multivariable model and produce a forest plot (). For short-term AE, BMI [odds ratio (OR) =0.91], myocardial infarction (OR =2.62), CHADS2 score (OR =0.49), and bleeding history or predisposition, labile INR, elderly, drug/alcohol usage (BLED) score (OR =1.71) were the significant risk factors for the short-term AE.

Figure 3

Forest plot of multiple risk factors. OR, odds ratio; CI, confidence interval; BMI, body mass index; MI, history of myocardial infarction; age75, age over 75 years; CHADS2, congestive heart failure, hypertension, 75 years, diabetes, stroke/TIA, vascular disease, age 65 to 74 years, gender category (women); Hyt, hypertension; BLED, bleeding; Hep, liver injury; Ked, kidney disease; INR, international normalized ratio; Alc, drinking history; DM, diabetes mellitus; CI, cerebral infarction; Blood, bleeding history.

After the exploratory analysis, we conducted further analysis of fit for 8 kinds of machine learning models (). In order to verify the validity of the hypothesis obtained in this study, we hypothesized hypothetical hypothesis. As a result, it is a test of the null hypothesis that all the coincidence is caused by the pure probability. And we were quantifying the ability for various models calculated by the machine learning algorithm, so it was a test of re discrimination. Next, the accuracy of each model is evaluated by counting how the number of predicted records coincides with the actual number of records. The comparison between 8 algorithms was based on accuracy of prediction and Kappa value (this value represents the discrepancy between the observed probability of success and the probability of success under the assumption of an extremely bad case). From the comparison, we found that the XGBoost model was the best choice for determining the relationship between LAAC and AE.

Figure 4

Comparison of 8 machine learning models for short-term AE. AE, adverse events.

We tested the influence of boosting for the model fitting, and the results showed that the model was steady. From iteration times 1 to 10, the accuracy of the model stayed at nearly 0.95 (). The XGBoost results also produced the variable importance plot (). In addition, a ROC curve was made to evaluate the prediction efficacy of XGBoost, which showed that area under the curve (AUC) was 0.85, indicating that the prediction of XGBoost was good.

Figure 5

Accuracy of XGBoost in different iterations.

Figure 6

ROC curve of XGBoost model for short-term AE. AUC, area under the curve; ROC, receiver operating characteristic; AE, adverse events.

To further analyze the long-term AE of LAAC surgery, we conducted survival analysis and a cumulative incidence plot. From the Cox regression analysis, 11 factors had significance including CHADS2 score [hazard ratio (HR) =1.43], hypertension (HR =2.18), age more than 75 years (HR =0.49), diabetes (HR =0.57), BLED score (HR =0.28), stroke (HR =19.8), hepatopathy (HR =3.97), nephropathy (HR =2.93), INR instability (HR =4.18), alcohol (HR =2.67), and drug use (HR =2.36). In the evaluation of the Cox-regression model, 3 methods were applied. The results showed that P values of likelihood ratio test, Wald test, and log-rank test were all less than 0.05, which demonstrated the significance of the model fitting.

After fitting XGBoost model, a cumulative incidence plot was made to predict the risk of long-term AE. After 100 days of receiving LAAC, the risk ratio tended to be stable, and the incidence of long-term AE was between 12% and 15%. The incidence of no long-term side effects remained at more than 80% ().

Figure 7

Plot in cumulative incidence of long-term AE. AE, adverse events.

Discussion

The results of this study showed that in the difference analysis of baseline data, except for the significant difference in BMI, there was no difference in other demographic data and risk factors affected by short-term complications. This is different from the previously reported results, which may be because the included samples were from a single center and there were some confounding factors. After univariate analysis, we conducted multivariate analysis and drew a forest map. From the results, we could see that in addition to the statistically significant difference in BMI (OR =0.91), CHADS2 score (OR =0.49) and BLED score (OR =1.71) were correlated in the occurrence of short-term complications. The BLED score had the greatest impact on the occurrence of short-term depression, and CHADS2 score was significantly correlated with depression, which is consistent with other reports (13,14).

After single factor and multi factor difference analysis, we fitted 8 machine learning models and 15 risk factors, and compared the accuracy and kappa values of the models. The XGBoost (xgbdart) algorithm had the best performance. We then used the XGBoost on the training data. The graph of iteration times showed that that the accuracy of the model remained stable with the change of iteration times, indicating that the training of the model was very good. On this basis, we put the 8:2 test set into the model, analyzed the predicted and actual values, and obtained the ROC diagram; the AUC =0.88, indicating that the model fit well. Therefore, the model can be used to predict short-term risk factors. In other words, the results of short-term complications can be predicted by inputting 15 risk factor values into the model.

In addition to analyzing and predicting short-term complications, we also analyzed the survival rate of long-term complications and constructed a Cox regression model. For long-term complication data, we used the first follow-up data included in the sample. The first follow-up ranged from 15 to 498 days. From the Cox regression model, we saw that CHADS2 score (HR =1.43), hypertension (HR =2.18), age over 75 years (HR =0.49), diabetes mellitus (HR =0.57), bleeding score (HR =0.28), stroke (HR =19.8), liver disease (atherosclerosis), nephropathy, INR instability, and drinking were significantly associated with long terms AE. Drug therapy (DRT; HR =2.36) was identified as an important influencing factor of long-term complications, which is very consistent with the previous research conclusions (14,15). The results showed that stroke, INR instability, liver disease, kidney disease, and drinking were the top 5 risk factors. Studies have shown that DRT is associated with a higher risk of composite end points of death, ischemic stroke, or systemic embolism. Multivariate analysis revealed 5 risk factors for DRT: hypercoagulable state, pericardial effusion, renal insufficiency, implantation depth >10 mm at the edge of pulmonary vein, and non-paroxysmal AF (2,8).

Previous studies have also suggested that clinical risk scores used to predict thromboembolism in AF, such as CHADS2, can be used to guide anticoagulation strategies (1,15). In addition, several blood biomarkers have been shown to be associated with the presence of LAA thrombosis. Brain natriuretic peptide (BNP), mean erythrocyte volume, mean platelet volume, and eosinophil count have additional diagnostic effects above CHADS2 (16,17). Clinical risk factors included hypertension, congestive heart failure, age, female, structural heart disease or cardiomyopathy, use of antiarrhythmic drugs, persistent and paroxysmal atrial fibrillation, and higher CHADS2 score (18,19). In addition to demographic and clinical risk factors, it is reported that the measurement of left atrial deformation parameters by two-dimensional speckle tracking can predict the impairment of left atrial function and the presence of left atrial thrombosis in patients with sinus rhythm and suspected cardiogenic stroke (20,21).

Overall, univariate and multivariate analyses were used to explore the influencing factors that may be related to short-term complications, mainly including BMI, CHADS2 score and BLED score. Among the complications of long-term follow-up, stroke, INR instability, liver disease, kidney disease, and alcohol consumption were the top 5 important factors. Of course, the impact of CHADS2 score cannot be ignored. The XGBoost model had a good performance in predicting AE after LAAC.

This study had several limitations. Firstly, it was a single center cohort study without randomized grouping and a control group. Although our results are comparable with those of other clinical trials, these conclusions must be carefully interpreted in the absence of a matched control group. Secondly, there were differences in the prescription rates of warfarin, antiplatelet drugs, and novel oral anticoagulants (NOACs) between early and late perioperative inpatients, which may affect the clinical efficacy. Thirdly, the ultrasonic measurement and interpretation by the operator were realized without independent image judgment. Finally, the number of patients included in this study was relatively small. In conclusion, for short- and long-term AE, CHADS2 scores and BLED scores were the most obvious risk factors. Several other risk factors also played a role in AE. The overall incidence of long-term AE is less than 15%, and LAAC is effective and safe.

The article’s supplementary files as