Background: Whether prophylactic central lymph node dissection is necessary for cN0 papillary thyroid microcarcinoma (PTMC) patients remains highly debatable. Surgeons desperately need a way to help with surgical decision-making. While traditional predictive models can better explain changes in variables, machine learning (ML) models may have better predictive performance. This study aims to develop models for predicting the risk of central lymph node metastasis (CLNM) by utilizing ML algorithms. Methods: The clinical records of 1,121 patients with cN0 PTMC who underwent initial thyroid resection at our hospital between January 2014 and December 2018 were retrospectively retrieved. Univariate and multivariate analyses were performed to examine risk factors associated with CLNM. Six ML algorithms for predicting CLNM were established and internally validated. Indices including the area under the receiver operating characteristic (AUROC), sensitivity, specificity, accuracy, positive predictive value (PPV), and negative predictive value (NPV) were calculated to test the performance of the model. Results: The results showed 33.5% (376 out of 1,121) of patients had CLNM. In multivariate logistic regression (LR) analyses, gender, age, tumor size, multifocal lesions, and extrathyroidal extension (ETE) were all independent predictors of CLNM. The AUROC predictive values of the six ML algorithms were between 0.664 and 0.794, with the random forest (RF) model performing the best with an AUROC of 0.794. Therefore, we used the RF model and uploaded the results to a web-based risk calculator to predict an individual's probability of CLNM (https://xijing-thyroid.shinyapps.io/ptmc_clnm). Conclusions: Developing predictive models of CLNM in cN0 PTMC patients using the ML algorithm is a feasible method. Our online risk calculator based on the RF model may be a useful tool for surgical decisions. 2022 Annals of Translational Medicine. All rights reserved.
Background: Whether prophylactic central lymph node dissection is necessary for cN0 papillary thyroid microcarcinoma (PTMC) patients remains highly debatable. Surgeons desperately need a way to help with surgical decision-making. While traditional predictive models can better explain changes in variables, machine learning (ML) models may have better predictive performance. This study aims to develop models for predicting the risk of central lymph node metastasis (CLNM) by utilizing ML algorithms. Methods: The clinical records of 1,121 patients with cN0 PTMC who underwent initial thyroid resection at our hospital between January 2014 and December 2018 were retrospectively retrieved. Univariate and multivariate analyses were performed to examine risk factors associated with CLNM. Six ML algorithms for predicting CLNM were established and internally validated. Indices including the area under the receiver operating characteristic (AUROC), sensitivity, specificity, accuracy, positive predictive value (PPV), and negative predictive value (NPV) were calculated to test the performance of the model. Results: The results showed 33.5% (376 out of 1,121) of patients had CLNM. In multivariate logistic regression (LR) analyses, gender, age, tumor size, multifocal lesions, and extrathyroidal extension (ETE) were all independent predictors of CLNM. The AUROC predictive values of the six ML algorithms were between 0.664 and 0.794, with the random forest (RF) model performing the best with an AUROC of 0.794. Therefore, we used the RF model and uploaded the results to a web-based risk calculator to predict an individual's probability of CLNM (https://xijing-thyroid.shinyapps.io/ptmc_clnm). Conclusions: Developing predictive models of CLNM in cN0 PTMC patients using the ML algorithm is a feasible method. Our online risk calculator based on the RF model may be a useful tool for surgical decisions. 2022 Annals of Translational Medicine. All rights reserved.
Entities:
Keywords:
Thyroid cancer; central lymph node metastasis (CLNM); machine learning (ML); microcarcinoma; model
Thyroid cancer is the most pervasive malignant endocrine carcinoma worldwide, the incidence of which has significantly increased over the past decades (1-3). This surge is mainly due to the rising incidence of papillary thyroid cancer (PTC) (4), particularly papillary thyroid microcarcinomas (PTMC), which are defined as thyroid cancers of 1 cm or less in size (5). Although PTMC is viewed as an indolent tumor, central lymph node metastasis (CLNM) is significantly detected in PTMC patients (6,7).According to the American Association of Endocrine Surgeons guidelines, therapeutic central compartment LN dissection (CLND) is recommended during initial thyroidectomy for PTMC patients with a clinical or image-detected nodal disease. However, implementing prophylactic CLND for low-risk PTMC remains controversial (8). The efficacy of preoperative neck ultrasound (US) to detect CLNM is limited due to the overlying thyroid gland (9). For PTMC, whether prophylactic CLND is performed during thyroidectomy often depends on the patient’s situation and the surgeon’s expertise. It is important to avoid overtreatment of prophylactic CLND because of surgical complications, including hypoparathyroidism and recurrent laryngeal nerve injury. On the other hand, simple thyroidectomy may contribute to disease recurrence (10), and second surgery is very difficult for cervical lymph node dissection. In recent years, some studies on the influencing factors of lymph node metastasis (LNM) have not excluded factors that have indications for lymph node resection, such as lateral LNM (7,11-14). Moreover, the conclusions of these studies are inconsistent. Therefore, it is clinically crucial to establish a reliable prediction model and identify CLNM before surgery for those patients who do not have strong surgical indications for lymphadenectomy.Machine learning (ML) was created in 1959 by Arthur Samuel, whose main purpose was to introduce algorithms that take in input data, capitalize on computer analysis to predict output values within an acceptable range of accuracy, identify trends and patterns within the data, and ultimately learn from previous experience (15). ML in medicine can conduce to more accurate diagnostic algorithms and individual patient treatment (15-17). Hence, for better prediction of CLNM and individualized treatment in cN0 PTMC patients, this study was designed to investigate the influencing factors of CLNM and develop promising prediction models using ML algorithms. We present the following article in accordance with the TRIPOD reporting checklist (available at https://atm.amegroups.com/article/view/10.21037/atm-22-3594/rc).
Methods
Patients
This retrospective study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the ethics committee board of Xijing Hospital (XJLL-KY-20220428) and individual consent for this retrospective analysis was waived. A total of 1,209 PTMC patients who underwent initial thyroidectomy at the Department of Endocrine Surgery between January 2014 and December 2018 were enrolled. PTMC is diagnosed by preoperative ultrasonography and fine needle aspiration. Inclusion criteria were patients with preoperative diagnosis of PTMC. The exclusion criteria were as follows: Incomplete medical information, other types of thyroid carcinomas, absence of CLND, and evidence of central compartment LNM detected by preoperative US. Finally, 1,121 patients were included in the study.
Surgical strategy
In our institution, surgical decisions are made according to the American Thyroid Association guidelines. Lobectomy plus isthmusectomy with ipsilateral CLND was performed for patients with unilateral lobe PTC, and total thyroidectomy with bilateral CLND was performed for patients with bilateral PTC or PTC in the isthmus. A total of thyroidectomy plus ipsilateral CLND was performed for patients with extrathyroidal extension (ETE) tumor or multifocal tumors limited to a unilateral lobe.
Data collection
Clinical characteristics collected from clinical and pathological records included gender, age, family history of cancer, US characteristics, thyroid function test, and clinicopathological features. A family history of cancer was defined as any cancer occurring among the first- and second-degree relatives. US information included maximal tumor size, microcalcification, irregular shape, and ill-defined margin, while thyroid function test results included free tetraiodothyronine (FT4), free triiodothyronine (FT3), tetraiodothyronine (T4), triiodothyronine (T3), and thyroid-stimulating hormone (TSH). Clinicopathological features included Hashimoto thyroiditis (HT), bilateral lesions, multifocal/solitary lesions, capsule invasion, primary tumor ETE, and BRAFV600E mutation. Cervical US and thyroid function test were performed within one month before the surgery, and clinicopathological features were collected from intraoperative frozen section and postoperative pathological results. All patients were diagnosed with PTMC pre-operatively by fine-needle aspiration biopsy or intraoperatively using the frozen section procedure.
Statistical analyses
R software version 4.0.5 (R Foundation for Statistical Computing, Vienna, Austria) was used for statistical analyses. The Shapiro-Wilk test was used to test the normal distribution of quantitative data, and for non-normally distributed data, the median with interquartile range (IQR) was applied for expression. The student’s t-test was used to analyze quantitative parameters and the Pearson’s chi-square test was used to analyze discrete parameters. Multivariate logistic regression (LR) analysis with stepwise selection was used to assess the independent risk factors of CLNM.We randomly divided the data into the test set (70%) and validation set (30%), which were used for model establishment and evaluation, respectively. Six types of ML algorithms models with all variables were developed in our study: LR, multivariate adaptive regression splines (MARS), decision tree (DT), random forest (RF), extreme gradient boosting (XGBoost), and neural network (NNET). We then calculated the area under the receiver operating characteristic (AUROC), sensitivity, specificity, accuracy, positive predictive value (PPV), and negative predictive value (NPV) of ML algorithms in the validation set. Relative importance ranking list was generated for each ML algorithm, and for the best-performing model, we calculated the error of different numbers of variables by ten cross validation to find the optimal number of variables. A two-sided P value <0.05 was considered statistically significant.
Results
Demographics features
The clinicopathological characteristics of 1,121 PTMC patients are shown in . Among them, 878 were female (78.3%), and 243 were male (21.7%), and the patients’ age at the time of initial treatment ranged from 16 to 73 years (median age 45 years; IQR: 37–51 years). The median tumor size was 0.7 (IQR: 0.5–0.8) cm, and 122 patients (10.9%) had a family history of cancer. Microcalcification was observed in 543 patients (48.4%) and an irregular tumor shape was detected in 819 (73.1%). A total of 798 patients (71.2%) had an ill-defined margin.
Table 1
Demographic and clinicopathological variables of the whole cohort grouped by lymph node status
IQR, interquartile range; FT4, free tetraiodothyronine; FT3, free triiodothyronine; T4, tetraiodothyronine; T3, triiodothyronine; TSH, thyroid stimulating hormone; ETE, extrathyroidal extension; CLNM, central lymph node metastasis.The median of FT4, FT3, T4, T3, and TSH were 16.5 (IQR: 15.1–18.1) pmol/L, 4.8 (IQR: 4.4–5.2) pmol/L, 104.8 (IQR: 93.0–117.1) nmol/L, 1.8 (IQR: 1.6–2.0) nmol/L, and 2.2 (IQR: 1.5–3.2) µIU/mL, respectively. The number of patients with HT was 170 (15.2%), and 310 (27.7%) had bilateral lesions, and 447 (39.9%) had multifocal lesions. Capsule invasion was present in 66 cases (5.9%), and primary tumor ETE was observed in 53 (4.7%). BRAFV600E mutation was detected in 799 patients (71.3%).
Univariate and multivariate analysis
Univariable analysis () showed gender, age, tumor size, microcalcification, FT3, and multifocality were all significantly associated with the occurrence of CLNM in PTMC patients (P<0.05). In multivariate logistic regression analyses (), seven variables (gender, age, family history of cancer, tumor size, ill-defined margin, multifocal lesions, primary tumor ETE) were selected using stepwise selection. The results showed that male gender (OR =2.06, 95% CI: 1.53–2.79, P<0.001), tumor size (OR =3.58, 95% CI: 1.86–6.88), multifocal lesions (OR =1.56, 95% CI: 1.19–2.03, P=0.001), and ETE (OR =1.85, 95% CI: 1.03–3.31, P=0.039) were independent positive predictors of CLNM, while older age (OR =0.96, 95% CI: 0.94–0.97, P<0.001) was a negative predictor.
Table 2
Univariate and multivariate logistic regression analysis of variables in predicting CLNM in whole cohort
All 18 variables were used in CLNM predictive models based on six ML algorithms models, whose prediction performance is shown in and . The RF model demonstrated the most potential for predicting CLNM in PTMC patients, showing the highest AUROC (0.794, 95% CI: 0.744–0.844) and highest accuracy (0.772). Threshold values of AUC in RF model is 0.423 and the sensitivity and specificity of this model were 0.564 and 0.872. Consequently, the RF model was selected as the prediction model in this study.
Figure 1
ROC curve analysis of machine learning algorithms for prediction of CLNM patients with cN0 PTMC in the validation set. LR, logistic regression; MARS, multivariate adaptive regression splines; DT, decision tree; RF, random forest; XGBoost, extreme gradient boosting; NNET neural network; ROC, receiver operating characteristic; CLNM, central lymph node metastasis; PTMC, papillary thyroid microcarcinoma.
Table 3
Predictive performance comparison of the six types of machine learning algorithms in the validation sets
Methods
AUROC (95% CI)
Sensitivity
Specificity
Accuracy
PPV
NPV
LR
0.709 (0.650–0.765)
0.664
0.674
0.671
0.497
0.805
MARS
0.668 (0.608–0.728)
0.773
0.520
0.602
0.438
0.825
DT
0.664 (0.601–0.728)
0.664
0.643
0.650
0.474
0.798
RF
0.794 (0.744–0.844)
0.564
0.872
0.772
0.681
0.805
XGBoost
0.699 (0.642–0.756)
0.700
0.595
0.629
0.456
0.804
NNET
0.678 (0.619–0.736)
0.791
0.515
0.605
0.442
0.836
LR, logistic regression; MARS, multivariate adaptive regression splines; DT, decision tree; RF, random forest; XGBoost, extreme gradient boosting; NNET, neural network; AUROC, area under the receiver operating characteristic; PPV, positive predictive value; NPV, negative predictive value.
ROC curve analysis of machine learning algorithms for prediction of CLNM patients with cN0 PTMC in the validation set. LR, logistic regression; MARS, multivariate adaptive regression splines; DT, decision tree; RF, random forest; XGBoost, extreme gradient boosting; NNET neural network; ROC, receiver operating characteristic; CLNM, central lymph node metastasis; PTMC, papillary thyroid microcarcinoma.LR, logistic regression; MARS, multivariate adaptive regression splines; DT, decision tree; RF, random forest; XGBoost, extreme gradient boosting; NNET, neural network; AUROC, area under the receiver operating characteristic; PPV, positive predictive value; NPV, negative predictive value.
Variable importance
The relative importance of the ten most important predictors of the six models is summarized in . Notably, age, tumor size, gender, and multifocality were ranked in the top 10 of all six models. The errors of RF reached the lowest when all 18 variables were introduced ().
Figure 2
Relative importance ranking of each variable for prediction of CLNM in the machine learning algorithms. (A) Logistic regression. (B) Multivariate adaptive regression splines. (C) Decision tree. (D) Random forest. (E) Extreme gradient boosting. (F) Neural network. LR, Logistic regression; MARS, multivariate adaptive regression splines; DT, decision tree; RF, random forest; FT4, free tetraiodothyronine; FT3, free triiodothyronine; T4, tetraiodothyronine; T3, triiodothyronine; TSH, thyroid stimulating hormone; ETE, extrathyroidal extension; CLNM, central lymph node metastasis.
Figure 3
Cross validation errors of the random forest model with different numbers of variables. The error was the lowest (0.304) with 18 variables.
Relative importance ranking of each variable for prediction of CLNM in the machine learning algorithms. (A) Logistic regression. (B) Multivariate adaptive regression splines. (C) Decision tree. (D) Random forest. (E) Extreme gradient boosting. (F) Neural network. LR, Logistic regression; MARS, multivariate adaptive regression splines; DT, decision tree; RF, random forest; FT4, free tetraiodothyronine; FT3, free triiodothyronine; T4, tetraiodothyronine; T3, triiodothyronine; TSH, thyroid stimulating hormone; ETE, extrathyroidal extension; CLNM, central lymph node metastasis.Cross validation errors of the random forest model with different numbers of variables. The error was the lowest (0.304) with 18 variables.
Web-based calculator
An online calculator based on the RF model was established to predict a patient’s risk of CLNM by simply entering clinical variables ().
Figure 4
Online calculator for predicting central lymph node metastasis in cN0 PTMC. FT4, free tetraiodothyronine; FT3, free triiodothyronine; T4, tetraiodothyronine; T3, triiodothyronine; TSH, thyroid stimulating hormone; ETE, extrathyroidal extension; HT, hashimoto thyroiditis; CLNM, central lymph node metastasis; PTMC, papillary thyroid microcarcinoma.
Online calculator for predicting central lymph node metastasis in cN0 PTMC. FT4, free tetraiodothyronine; FT3, free triiodothyronine; T4, tetraiodothyronine; T3, triiodothyronine; TSH, thyroid stimulating hormone; ETE, extrathyroidal extension; HT, hashimoto thyroiditis; CLNM, central lymph node metastasis; PTMC, papillary thyroid microcarcinoma.
Discussion
The incidence of PTMC has increased rapidly in the last few decades (5). According to previous studies, CLNM is an important risk factor in the recurrence of PTMC but is rarely identified clinically (18-20). The study conducted by Qu et al. demonstrated CLNM prevalence ranged between 17.8% and 64.1% (6), while in this study, the CLNM incidence in cN0 PTMC was 33.5% (376/1,121). Although it is widely accepted that CLND should be performed in patients with positive clinical CLNM, for PTMC patients without preoperative and intraoperative suspected CLNM, the need for CLND remains controversial (21,22) and mostly relies on the surgeon’s experience. Several studies have aimed to recognize predictive factors associated with CLNM in cN0 PTMC patients, but the results were inconsistent (23-25). Although ML algorithms are gradually being used for clinical decisions, few studies utilized them to establish a CLNM prediction model (16,17). Therefore, besides univariate and multivariate analysis, we developed and internally validated multiple models for predicting CLNM in cN0 PTMC patients using ML algorithms.Herein, a retrospective cohort of 1,121 PTMC patients was assessed. A total of 18 variables, including clinicopathological features and US characteristics, were collected for analysis and constructing prediction models, and multivariate analysis showed tumor size, male, young age, multifocality, and ETE were highly predictive of CLNM. This is similar to several previous studies (11,26), while it was not completely consistent with some reports. Siddiqui’s et al. revealed that male and tumor size were not related to CLNM (27), while in Yang et al.’s study, age had no significant correlation with CLNM (28). Microcalcification, FT3, and T3 correlated with CLNM in the univariate analysis but had no significance after cofounding adjustment in multivariate analysis. Recent research verified BRAFV600E mutation was associated with CLNM (23,28,29), but Kim et al. reported it was not an indicator of CLNM (30). In our study, no significant correlation was seen between BRAF mutation and CLNM.Compared with previous studies predicting the risk of CLNM in PTMC (7,11), our work focused on cN0 PTMC patients. Although ML in medicine has gradually become a hot topic in recent years (31), to the best of our knowledge, this is the first study to develop a prediction model based on ML algorithms for real-time risk evaluation of CLNM in cN0 PTMC patients. Herein, we used six different ML algorithms to establish CLNM prediction models and show the importance of variables in them. Although the importance of variables in different models was not completely consistent, age, tumor size gender, and multifocal lesions were always highly ranked. Some variables, such as FT4, T3, and T4, showed no significance in univariate or multivariate analysis but occupied important positions in some ML models. According to AUROC values (0.794) and the accuracy (0.772) of the models, RF was selected as the best predictive model, and has achieved better performance than traditional methods used by previous studies (23,32,33). Furthermore, to make this ML-based model easier to use, we built an online application based on it, and clinicians can now promote personalized surgical treatment by calculating the risks for each patient.There are several limitations to this study. Firstly, our work is a single-center retrospective observational study. Hence the effectiveness of the models has only been verified internally, and a potential selection bias exists. Secondly, some variables in the models are based on frozen biopsy and postoperative pathological diagnosis, which might restrict the ability to detect CLNM before surgery. From our point of view, pathological results are more stable and consistent than US results. The clinicopathology features in our model may be collected by fine-needle aspiration and frozen sections before or during surgery in the future. Thirdly, our study lacks long follow-up data to verify the prognosis of the patients. Finally, calculated according to the cut-off value we gave, the RF model has high accuracy but low sensitivity. However, this model still has high potential clinical application value. Obviously, when the accuracy is certain, the sensitivity and accuracy cannot reach a high level at the same time. Although this model has low sensitivity, it has a high PPV. Therefore, considering that surgery may cause damage to nerves or the thoracic duct, for surgical teams lacking nerve detectors, after evaluated by this predictive model, patients with higher predictive values were offered prophylactic lymphadenectomy. In patients with low levels, the surgeon can make an intraoperative decision whether to prophylactically remove the lymph nodes. Despite these shortcomings, our study is the first to apply ML algorithms to CLNM prediction in cN0 PTMC patients and may offer a basis for future clinical treatment.
Conclusions
This study developed and validated ML algorithms for individualized risk prediction of CLNM in cN0 PTMC patients. Our ML-based prediction model and its accompanying online risk calculator can provide clinicians and researchers with a simple tool to make precise surgical decisions. In the future, we aim to incorporate multiple biological variables, including imaging, molecular, genetic, and immune microenvironment data, to further improve the model’s performance.The article’s supplementary files as
Authors: Ian D Hay; Maeve E Hutchinson; Tomas Gonzalez-Losada; Bryan McIver; Megan E Reinalda; Clive S Grant; Geoffrey B Thompson; Thomas J Sebo; John R Goellner Journal: Surgery Date: 2008-12 Impact factor: 3.982
Figure 1
Figure 2
Figure 3
Figure 4