Machine Learning for the Prediction of Lymph Nodes Micrometastasis in Patients with Non-Small Cell Lung Cancer: A Comparative Analysis of Two Practical Prediction Models for Gross Target Volume Delineation.

PURPOSE: The lymph node gross target volume (GTV) delineation in patients with non-small cell lung cancer (NSCLC) is crucial for prognosis. This study aimed to develop a predictive model that can be used to differentiate between lymph nodes micrometastasis (LNM) and non-lymph nodes micrometastasis (non-LNM). PATIENTS AND METHODS: A retrospective study involving 1524 patients diagnosed with NSCLC was collected in the First Hospital of Wuhan between January 1, 2017, and April 1, 2020. Duplicated and useless variables were excluded, and 16 candidate variables were selected for further analysis. The random forest (RF) algorithm and generalized linear (GL) algorithm were used to screen out the variables that greatly affected the LNM prediction, respectively. The area under the curve (AUC) was compared between the RF model and GL model.
RESULTS: The RF model revealed that the variables, including pathology, degree of differentiation, maximum short diameter of lymph node, tumor diameter, pulmonary membrane invasion, clustered lymph nodes, and T stage, were more significant for LNM prediction. Multifactorial logistic regression analysis for the GL model indicated that vascular invasion, tumor diameter, degree of differentiation, pulmonary membrane invasion, and maximum standard uptake value (SUVmax) were positively associated with LNM. The AUC for the RF model and GL model was 0.83 (95% CI: 0.75 to 0.90) and 0.64 (95% CI: 0.60 to 0.70), respectively.
CONCLUSION: We successfully established an accurate and optimized RF model that could be used to predict LNM in patients with NSCLC. This model can be used to evaluate the risk of an individual patient experiencing LNM and therefore facilitate the choice of treatment.

Introduction

Globally, lung cancer cases and deaths remain rising over the past decade. In 2018, the Global Cancer Observatory Organization (GLOBOCAN) estimated more than 2 million new cases and 1.76 million deaths, obviously higher than 2012 reported rates (1.8 million new cases and 1.6 million deaths).1 NSCLC has been always the most common cause of morbidity and mortality worldwide.2 Over the past two decades, important advancements in the treatment of NSCLC have been achieved, such as chemotherapy, molecular-targeted therapy, or immunotherapy.3 However, the overall cure and survival rates for NSCLC remain low, particularly in metastatic disease.3 Therefore, continued research into the optimization of treatment strategy is urgent.

Mediastinal and hilar LNM is one of the recurrence patterns after definitive treatment of lung cancer.4 Lymph node dissection is an important part of this procedure that can improve the prognosis of the patients diagnosed with early stage.5 Although radiotherapy has proven to be effective for LNM, radiation pneumonitis inevitably decreases patients’ performance status and quality of life, which can be related to a worse long-term prognosis.6 Hence, evaluation of regional LNM is important for the clinician to determine the GTV delineation and prognosis.7 Especially, repeated radiotherapy for lymph node recurrence after primary tumors should be evaluated carefully, because irradiation areas are wide and an excess radiation dose can be caused by overlapping radiotherapy.4,8 Additionally, hilar and mediastinal lymph node metastases are frequently located close to organs at risk.4 Therefore, it is an urgent need to develop an accurate model for predicting the risk of LNM that can be used to facilitate the management of clinical treatment.

The rapid evolution of knowledge in artificial intelligence, a new machine learning choice that predicts the risk of LNM with better accuracy has led some clinicians to question the traditional predictive linear model in clinical care at an individualized patient level.7 The RF algorithm allows a machine to be fed with raw data and to automatically screen the useful variables needed for detection with multiple neural layers in the network.9,10 Ensemble predictors such as the RF model are known to have superior accuracy. Besides, a GL model is very interpretable especially when forward feature selection is used to construct a predictive model.11

In the present study, we aimed to develop a predictive model via machine learning that can be used to differentiate between LNM and non-LNM. The validated capability of enabling expeditious and accurate LNM risk stratification of NSCLC may facilitate more responsive radiotherapy that is conducive to high-risk NSCLC patients via early identification, and ensuing instant intervention as well as precise GTV delineation and monitoring, thus, hopefully assisting to manage therapeutic intervention more appropriately.

Patients and Methods

Patients Enrollment

Between January 1, 2017, and April 1, 2020, we retrospectively collated data from consecutive patients who had been diagnosed with NSCLC at the First Hospital of Wuhan. All patients who had received lobectomy or segmentectomy with complete lymph node dissection as per the nomenclature were selected for this study. This study was approved by the Institutional Ethics Committee of the Wuhan No.1 Hospital (Reference: [2020] No.31), in compliance with the Declaration of Helsinki. Written informed consent was obtained from all participants before any treatment. All patients’ information was anonymous. The inclusion criteria were as follows: (1) patient had undergone oncological surgical resections with systematic nodal dissection; (2) patients’ TNM classification (Until then, 6th and 7th edition) were transferred into the definitions of 8th edition of American Joint Committee on Cancer (AJCC) by two pathologists to form unified tumor stage classification. Exclusion criteria were as follows: (1) patient received any neoadjuvant therapy; (2) patient with limited and/or palliative resections; (3) patient with carcinoid tumors.

Evidence of Lymph Node Resection and Staging

Systematic nodal dissection was performed according to the technique as described.12 The mediastinal lymph nodes were dissected systematically within anatomical landmarks and labeled as described according to Mountain&Dresler map (Until then, 2010),13 and thereafter followed the guideline of the International Association for Study of Lung Cancer (IASLC).14 Clinical staging was based on the results of blood tests, Thorax-CT, PET/CT, Brain MRI (mainly for symptomatic patients), and Bronchoscopy. Patients with cN2, cN3 according to the radiological findings received invasive mediastinal staging via the Endobronchial Ultrasound (EBUS) findings. LNM status preoperatively was evaluated as follows: (1) Negative lymph nodes, calcification, and fat in the lymph node; (2) Positive lymph nodes, the short axis of the lymph node ≥10 mm, necrosis in the lymph nodes.15

Data Preparation

Among 46 original variables, we excluded those duplicated variables via correlation matrix analysis. The candidate variables such age, gender, degree of differentiation, histological tumor type, tumor size, and tumor site. Furthermore, the histological and morphological parameters such as pT-category, SUVmax, vascular invasion, pulmonary membrane invasion, clustered lymph nodes, and the maximum short diameter of lymph node.16 We also examined the localization and lobe distribution as to the potential effect on LNM in patients with NSCLC.

Statistical Analysis and Evaluation of Models

For descriptive analysis, continuous variables were presented as the mean with standard deviation (SD). Categorical variables are presented as numbers (%). Bonferroni correction adjusts probability values were used to compare qualitative data.17 The random forest was developed from various decision tree models.18 Random allocated variables were used to evaluate the random forest prediction model. The importance of candidate variables was mirrored by the mean decreased Gini(MDG) score. Besides, cross-validation is performed in the random forest algorithm, which can be merely as useful as depending on a separate test set to evaluate the generalization error of the training set. The stepwise regression based on the Akaike information criterion (AIC) minimum was used to select variables for inclusion in the GL model. The GL model was used to evaluate direct and indirect associations between LNM and candidate variables.19 Concordance index (C-index) and area under the curve (AUC) calculated by bootstrapping plots were used to evaluate calibrating ability. C-index and AUC values vary from 0.5 to 1.0, where 0.5 represents random chance and 1.0 indicates a perfect fit. Typically, C-index and AUC values greater than 0.7 suggest a reasonable estimation. All analyses were performed using the Python programming language (version 3.9.2, Python Software Foundation, ) and R Project for Statistical Computing (version 4.0.4, ). All P values were two-tailed, and P<0.05 was considered statistically significant.

Results

Patient Demographics and Characteristics

A total of 1524 patients diagnosed with NSCLC were collected in this study. A correlation matrix was spotted, and 16 candidate variables correlated with LNM are summarized in Figure 1. The whole data set was randomly and automatically divided into a training set (N=991,65%) and a testing set (N=533, 35%). Additionally, the systematic nodal dissection with removal of all ipsilateral hilar and mediastinal lymphatic tissue was carried out for all patients in oncological lung resection. All patients with NSCLC were diagnosed by imaging and pathology, and a total of 5196 groups of mediastinal and hilar small lymph nodes were obtained. Among them, 816 groups were pathologically confirmed, metastasis group. The pathological positive rate of the small lymph node group was 15.7%, mainly distributed in hilar lymph nodes (34.4%), inferior paratracheal lymph nodes (25.1%), and subcarinal lymph nodes (16.2%). The detailed patient demographics and characteristics are shown in Table 1.

Figure 1

Correlation matrix of candidate features. Values in this matrix demonstrated the correlation coefficient of each corresponding variable.

Table 1

Patients’ Demographics and Clinicopathological Characteristics

Variables	No. (%)
	(N=1524)
Age,year
Mean	46.8
SD	12.1
Sex
Male	1014 (66.5%)
Female	510 (33.5%)
Smoking
Yes	983 (64.5%)
No	541 (35.5%)
Site
Central type	633 (41.5%)
Peripheral type	891 (58.5%)
BMI,kg/m2
>27	407 (26.7%)
≤27	1117 (73.3%)
Differentiation
Poorly differentiation	180 (11.8%)
Moderately differentiation	839 (55.1%)
Highly differentiation	505 (33.1%)
Pathology
Squamous carcinoma	946 (62.1%)
Adenocarcinoma	539 (35.4%)
Others	39 (2.6%)
SUVmax
≤8	1175 (77.1%)
>8	349 (22.9%)
T stage
T1	278 (18.2%)
T2	1121 (73.6%)
T3	87 (5.7%)
T4	38 (2.5%)
Tumor Diameter,cm
>3	481 (31.6%)
≤3	1043 (68.4%)
Vascular invasion
Yes	876 (57.5%)
No	648 (42.5%)
Pulmonary membrane invasion
Yes	433 (28.4%)
No	1091 (71.6%)
Clustered lymph nodes
Yes	893 (58.6%)
No	631 (41.4%)
Maximum short diameter of lymph nodes
>0.6	255 (16.7%)
≤0.6	1269 (83.3%)
KRAS mutation
Yes	176 (11.5%)
No	1348 (88.5%)
EGFR mutation
Yes	420 (27.6%)
No	1104 (72.4%)

Abbreviations: SD, Standard deviation; BMI, Body mass index; SUVmax, Maximum standard uptake value; T stage, tumor node metastasis classification (AJCC7th); AJCC, American Joint Committee on Cancer.

RF Model Building

A total of 991 patients from the training set were used to fit the random forest algorithm. The samples were randomly assigned into the non-overlapping training samples for establishing the RF prediction model (Figure 2A). Generally, the 16 candidate variables were ordered according to the MDG score (Table 1). The significant variables, such as pathology, degree of tumor differentiation, the maximum short diameter of lymph node, tumor diameter, pulmonary membrane invasion, clustered lymph nodes, and T stage, were more significant for LNM predictive model construction (Figure 2B). The results of internal validation by ten-fold cross-validation showed the stability of the model’s predictive power (Figure 2C).

Figure 2

Random forest model. (A) The candidate factors associated with micrometastasis of lymph nodes were ordered according to the mean decreased Gini index. (B) Relationship of dynamic changes between the prediction error and the number of decision trees. (C) Performance of the prediction model with increasing numbers of features in the tenfold cross-validation.

GL Model Building

The GL model was widely used to extract all neural responses inside an epoch.20 Adjusting for important prognostic factors or baseline covariates in generalized linear models may improve the estimation efficiency.21 Hence, through stepwise regression, the included candidate predictors and the corresponding associations were almost the same with the model developed among the AIC filter, such as tumor diameter, degree of tumor differentiation, pulmonary membrane invasion, SUVmax, and vascular invasion. Besides, the C-index and the Brier score were performed to predict the performance of each model (Table 2). According to the AIC filter, the stability and potential utility of model 2 was confirmed for GL model construction, including vascular invasion (OR: 0.31, 95% CI: 0.25–0.40), differentiation (Moderate vs High: OR: 0.97, 95% CI: 0.85–1.10; Poorly vs High: OR: 1.54, 95% CI: 1.39–1.72), tumor site (OR: 2.69, 95% CI: 1.96–3.69), EGFR (OR: 1.07, 95% CI: 0.94–1.22), and SUVmax (OR: 2.49, 95% CI: 2.18–2.84).

Table 2

The Predictive Performances of Different Models Associated with Micrometastasis of Lymph Nodes

GL Model	Multivariate Analysis		Discrimination
	OR(95% CI)	P-value	Brier	R2	C-Index	AIC
Model1
Sex(Male vs Female)	0.94 (0.87–1.02)	<0.01
Age*,y	1.11 (0.99–1.24)	0.11
Diameter,cm(≤3cm vs>3)	1.11 (0.99–1.24)	0.06
Vascular invasion(Yes vs No)	0.31 (0.24–0.39)	<0.01
Differentiation(Moderate vs High)	0.97 (0.85–1.10)	<0.01	0.05	0.06	0.681	328.60
(Poorly vs High)	1.55 (1.39–1.73)	<0.01
Pulmonary membrane invasion(Yes vs No)	2.67 (1.95–3.67)	<0.01
EGFR(Positive vs Negative)	1.01 (0.89–1.16)	0.83
SUVmax(≤8.0 vs>8.0)	2.59 (2.25–2.98)	<0.01
Model2
Vascular invasion(Yes vs No)	0.31 (0.25–0.40)	<0.01
Differentiation(Moderate vs High)	0.97 (0.85–1.10)	<0.01
(Poorly vs High)	1.54 (1.39–1.72)	<0.01
Tumor site(Central vs Peripheral)	2.69 (1.96–3.69)	<0.01	0.05	0.06	0.686	327.50
EGFR(Positive vs Negative)	1.07 (0.94–1.22)	0.30
SUVmax(≤8.0 vs>8.0)	2.49 (2.18–2.84)	<0.01
Model3
Differentiation(Moderate vs High)	0.92 (0.96–0.99)	<0.01
(Poorly vs High)	0.31 (0.24–0.39)	0.03
Vascular invasion(Yes vs No)	0.97 (0.85–1.10)	<0.01
Pulmonary membrane invasion(Yes vs No)	1.55 (1.39–1.72)	<0.01	0.04	0.07	0.685	330.60
SUVmax(≤8.0 vs>8.0)	2.68 (1.96–3.68)	<0.01
Diameter, cm(≤3cm vs >3)	2.62 (2.28–3.02)	<0.01

Note: *Continuous variable.

Abbreviations: AIC, Akaike information criterion; GL model, generalized linear model; OR, odds ratio; 95% CI, 95% confidence level; SUVmax, Maximum standard uptake value.

RF Model Robustness Verification in the Training and Testing Set

The random forest was commonly used to better distinguish the possibility of LNM in patients with NSCLC. The efficacy of the prediction model was assessed using the training set (Figure 3A) and testing set (Figure 3B). As is shown in the multidimensional classification chart of the random forest, which could better distinguish the patient with the possibility of LNM or not.

Figure 3

Validation and comparison of the predictive model. The training set (A) and testing set (B) associated with micrometastasis of lymph nodes were measured via the RF model. The comparison of the ROC curve of the RF model (C) and GL model (D).

Comparison of Two Predictive Models in the Testing Set

The accuracy of probabilistic predictions between the GL model (Figure 3C) and RF model (Figure 3D) was compared using the testing set. The AUC values were 0.83 (95% CI: 0.75 to 0.90) for the RF model and 0.64 (95% CI: 0.60 to 0.70) for the GL model, which presented the statistical significance of the two prediction models (P<0.05).

Discussion

Precise evaluation of LNM in patients with NSCLC is quite important for clinicians to determine the treatment and prognosis. Accordingly, the IASLC lymph node map is employed in the 8th edition of the TNM staging system.22 Based on the sequence of the IASLC lymph node map, ipsilateral interlobar lymph nodes were first affected, followed by hilar lymph nodes, and then mediastinal lymph nodes. Nagata et al reported that detail of patterns of failures after stereotactic body radiotherapy for stage I lung cancer, and the LNM was 14.2% and isolated LNM without other recurrent site was 3.0%.23 Gorai et al reported that pathological N1 and N2 were incidentally observed in 9.3% and 12.3% in clinical stage IA lung cancer patients who underwent pulmonary resection.24 Kelsey et al reported that Hilar and mediastinal LNM after surgery was detected in 7% of stage I to II lung cancer patients.25 Besides, isolated LNM can be curable by aggressive local treatment including surgery and RT, because the disease site is the only regional.4 In addition, the ideal treatment strategy for LNM remains uncertain. Surgery is the mainstay of LNM, but salvage surgery after the first surgery is not feasible due to the patient burden. Salvage radiotherapy can be a treatment option for LNM. However, the hilar and mediastinal LNM are frequently located close to organs at risk. Detection of disseminated tumor cells is a great technical challenge, and many different technologies have been developed to enhance the sensitivity and specificity of the testing for micrometastasis.26 As for the pathological diagnosis, IHC is the most common staining method used to diagnose suspected micrometastasis. However, micrometastasis is very hard to pathologically diagnose even in permanent sections, even on ample incisional biopsies or resection specimens. Therefore, it is important to establish the treatment planning methods for GTV delineation to avoid organ injury in radiotherapy. In this study, we established an LNM prediction model based on machine deep learning. Compared with traditional linear prediction models, the performance of the LNM prediction model obtained by machine learning is better. To our knowledge, this is the first GTV delineation-assisted LNM prediction model applying a deep learning method.

To date, due to the lack of a “gold standard” for GTV delineation, there is considerable controversy in the delineation of the GTV of the lymph node.27 The International Commission on Radiation Units and Measurements (ICRU) defined the GTV as a tumor area evaluated via clinical examination and imaging. Herth et al reported that Endobronchial ultrasound-guided transbronchial needle aspiration (EBUS-TBNA) contributed to sample enlarged mediastinal lymph nodes in patients with NSCLC.28 Haruki et al observed that having a micropapillary component was one of the significant predictors of unexpected node-positive diseases.29 Nomori et al summarized that although PET was superior to CT scanning in lymph node staging in lung cancer, PET was unable to distinguish metastatic foci smaller than 4 mm, which were not unusual sizes for lymph node metastases in lung cancer. Positive lymph nodes with PET than 9 mm are likely to be true-positive rather than false-positive.30 Collectively, accurate prediction of occult lymph node involvement based on associated risk factors is necessary, which might facilitate pretherapy evaluation and decision-making.

In this study, we developed an RF model for the LNM prediction. When applied clinically, this LNM prediction model might serve as a tool to select patients for precise GTV delineation or provide clinicians with an opportunity to select alternative treatment options. Unlike other established risk assessment methods, the RF model not confined to a small number of risk factors could incorporate all candidate risk factors. With the help of machine learning, we screened out some meaningful variables, including tumor diameter, degree of tumor differentiation, pulmonary membrane invasion, consistent with previous studies.29–32 Meanwhile, we also got some variables that were not developed before, including the maximum short diameter of lymph nodes and clustered lymph nodes. To date, there have been no prospective studies to evaluate the correlation between clustered lymph nodes. Our research demonstrated that there is a correlation between clustered lymph nodes and lymph node metastasis, and clusters of small lymph nodes have a higher risk of metastasis. In other cancer research, the definition of a positive lymph node is also different. In nasopharyngeal carcinoma, clusters of 3 or more lymph nodes with a minimum diameter greater than 0.8 cm are considered positive lymph nodes.33 However, in rectal cancer, 3 or more small lymph nodes less than 0.8 cm in clusters are considered metastatic lymph nodes.34 Our research also suggested that in terms of lymph node size, the risk of tumor metastasis is higher in the small lymph node group with the shortest diameter of the largest lymph node greater than or equal to 0.6 cm. Previous studies have also reported that the short diameter of the largest lymph node for small lymph node metastasis is between 0.6 and 0.9 cm, which is consistent with our research results.35,36 As a difficult point in non-invasive diagnosis, the best cutoff value appears to be more limited in a linear model. However, using a nonlinear model, we can ignore the cut-off value of the short diameter of the largest lymph node, thereby improving the flexibility and accuracy of prediction.

Several studies have demonstrated that the incidence of LNM differs according to individual clinical parameters and histologic components within the tumor.37–40 Appropriate targeted therapy is very effective in patients with advanced NSCLC who have specific genetic alterations.41 Besides, Dong et al reported that patients with high levels of pre-stereotactic body radiotherapy SUVmax had poorer overall survival and local control and higher distant metastases.42 In our correlation matrix analysis, we also found that there is indeed a significant correlation between these variables, which suggests that they can be used as potential predictors of LNM. Consequently, our RF model performs an iterative analysis of candidate variables and shows very robust results in predicting LNM.

Nevertheless, the present study has several limitations that need to be considered. This study was based on data from a single institution. Although our validation was robust, it is now necessary to conduct external validation using data from other centers. Second, the ideal treatment strategy for isolated lymph node metastasis remains uncertain, which might have changed the denominator of truly occult disease. Third, because the model parameters were based on clinical collectable variables, the application of potential variables (such as immunological diagnosis biomarkers, genetical analysis) might improve the accuracy of our prediction model. Future research should be cautious and validated carefully.

Conclusion

To conclude, combinatorial applications of the LNM risk prediction model for NSCLC and electronic health records with readily available features can enable timely and accurate risk stratification of NSCLC patients on admission. The RF model can potentially assist clinicians to promptly target the high-risk patients on admission, and accurately contribute to GTV delineation assessment of patients with NSCLC, although external validation is required.