Risk Assessment of Pulmonary Metastasis for Cervical Cancer Patients by Ensemble Learning Models: A Large Population Based Real-World Study.

OBJECTIVE: Pulmonary metastasis (PM) is an independent risk factor affecting the prognosis of cervical patients, but it still lacks a prediction. This study aimed to develop machine learning-based predictive models for PM.
METHODS: A total of 22,766 patients diagnosed with or without PM from the Surveillance, Epidemiology, and End Results (SEER) database were enrolled in this study. The cohort was randomly split into a train set (70%) and a validation set (30%). In addition, 884 Chinese patients from two tertiary medical centers were included as an external validation set. Duplicated and useless candidate variables were excluded, and sixteen variables were included for the machine learning algorithm. We developed five predictive models, including the generalized linear model (GLM), random forest model (RFM), naive Bayesian model (NBM), artificial neural networks model (ANNM), and decision tree model (DTM). The predictive performance of these models was evaluated by the receiver operating characteristic (ROC) curve and calibration curve. The Cox proportional hazard model (CPHM) and competing risk model (CRM) were also included for survival outcome prediction.
RESULTS: Of the patients included in the analysis, 2456 (4.38%) patients were diagnosed with PM. Age, organ-site metastasis (liver, bone, brain), distant lymph metastasis, tumor size, and pathology were the important predictors of PM. The RFM with 9 variables introduced was identified as the best predictive model for PM (AUC = 0.972, 95% CI: 0.958-0.986). The C-index for the CPHM and CRM was 0.626 (95% CI: 0.604-0.648) and 0.611 (95% CI: 0.586-0.636), respectively.
CONCLUSION: The prediction algorithm derived by machine-learning-based methods shows a robust ability to predict PM. This result suggests that machine learning techniques have the potential to improve the development and validation of predictive modeling in cervical patients with PM.

Introduction

Cervical cancer is the fourth most common cancer worldwide, with more than half a million new cases and 311 365 deaths in 2018.1,2 According to the US National Comprehensive Cancer Network and the International Federation of Gynecology and Obstetrics (FIGO) guidelines, radical hysterectomy, bilateral pelvic lymphadenectomy, and elective oophorectomy for patients with clinical stage are recommended.3,4 Women with early-stage cervical cancer who undergo a radical hysterectomy are usually cured, with 5-year disease-free survival (DFS) rates exceeding 90% in some studies.5–8 However, delay in the treatment of cancer with advanced stage can have adverse consequences on the outcome, improvements in the clinical stage and effective treatment have led to long-term survival among patients with cervical cancer.9

The possibility for cervical cancer patients to develop a subsequent primary cancer becomes a more important consideration for prognosis.10 An emerging issue in survivors is the occurrence of metastasis formation, which is the major cause of death in most cervical patients with primary cancer. Lymph node metastasis is the main metastatic pathway and the most critical factor for overall survival (OS) and DFS of cervical cancer cases.11 However, the lymph node status is not involved in the FIGO staging system of uterine cervical cancer, which leads to an absence of important information for prognosis and treatment. Besides, distant site metastasis such as pulmonary, subsequent liver, bone, and brain, are the most common metastasis in cervical cancer.12 Despite considerable advances in the treatment, patients with distant metastasis still suffer dismal prognosis.13 Most cervical patients present with asymptomatic, or diagnosed after follow-up visit via imaging examination, which leads to delays in treatment and adverse prognosis.14 Collectively, we lack standardized estimates of the risk of pulmonary metastasis (PM) for cervical cancer patients.

Machine learning is a branch of computer science and statistics that generates predictive or descriptive models, which has been widely used for data analysis in medicine during the past decade.15,16 Compared with the conventional linear regression model, the supervised machine learning algorithm has enabled the opportunity to move data science research towards translation for more personalized cancer treatments.17 This study aimed to develop and validate effective machine learning-based models for the prediction of PM in patients with cervical cancer.

Materials and Methods

Study Population

Patients with cervical cancer were identified from the SEER program of the National Cancer Institute (). The cohort was composed of patients pathologically confirmed with PM from the SEER database between 2010 and 2016. The accessible data were downloaded SEER*Stat 8.3.6. The primary site code was restricted to the cervix uteri (International Classification of Diseases for Oncology-3/WHO 2008). Since the SEER database began to record metastatic information including pulmonary metastasis from 2010 to 2016. We obtained the information in cervical patients with PM from 1 January 2010 to 31 December 2016. In addition, 884 Chinese patients from two tertiary medical centers (The Central Hospital of Enshi Tujia and Miao Autonomous Prefecture; Hubei Cancer Hospital) were included as an external validation set. This study followed the declaration of Helsinki. Written informed consent was not obtained because the SEER database was publicly available, the data were analyzed retrospectively and anonymously. The flowchart of the patients’ selection is listed in Figure 1.

Figure 1

The flow chart of selecting the cervical patients with PM.

Variables Preparation

A total of 12 clinical characteristics and pathological features were analyzed in this study. However, the variables that can be used for further analyses were including age, race, primary site, cervix tumor size, histological grade, TNM stage (7th ed), number of excised lymph nodes, the outcome of lymph node biopsy, surgery for the cervix or not, location of tumor metastasis, insurance, and marital status. In the SEER database, several methods were introduced to define race. We also redefined race as white, black, and others (American Indian/AK Native, Asian/Pacific Islander). The grade was defined as unstaged and low, moderate, or high differentiation. The insurance was defined as insured, uninsured, and any Medicaid. The candidate variables, except age, were defined as categorical variables. Overall survival (OS) was defined as the date of diagnosis to the date of death due to any cause or the date of the last follow-up. Cancer-specific survival (CSS) was calculated as the time from the date of diagnosis to the date of death attributed to cervical cancer or the date of the last follow-up.

Construction of Machine Learning-Based Models

All patients were randomly divided into training and testing groups at a ratio of 7 to 3. Five algorithms were applied to predict PM, including the generalized linear model (GLM), random forest model (RFM), naive Bayesian model (NBM), artificial neural networks model (ANNM), and decision tree model (DTM).17–19 Both the Cox proportional hazard model (CPHM) and competing risk model (CRM) were evaluated in this study.20,21

Feature Selection and Evaluation of Models

To minimize the negative influence of overfitting in machine learning algorithms, tenfold cross-validation (CV) was repeated ten times to evaluate the accuracy of models. To rank and select meaningful variables, the optimal variables in different algorithms were ranked, and the variables with high ranks were developed for the establishment of prediction models. Differences of models between predicted and actual data were also recorded. We also used the area under the curve (AUC) of the receiver operating characteristic (ROC) curve to evaluate the performances of different models.

Statistical Analysis

For descriptive analysis, continuous variables were presented as the mean with standard deviation or as median with interquartile range (IQR). Categorical variables are presented as numbers (%). The OS, CSS, and risk assessment of PM were estimated using the nomogram score. The nomograms were added across dependent variables to derive total points, which were converted to predicted probabilities. The predictive performance of the nomogram predictive model was measured by concordance index (C-index) and calibration with 1000 bootstrap samples to decrease the overfit bias. All analysis was performed using the Python programming language (version 3.9.2, Python Software Foundation, ) and R Project for Statistical Computing (version 4.0.4, ). All P values were two-tailed, and P<0.05 was considered statistically significant.

Results

Demographic and Clinicopathological Characteristics

A total of 22,766 cervical cancer patients diagnosed with PM or not were collected in this study. A correlation matrix was spotted, and sixteen candidate variables correlated with PM were summarized in . The whole data set was randomly and automatically divided into a training set (N=15936, 70%) and a testing set (N=6830, 30%). Comparing to patients without pulmonary metastasis, the impacts of age, race, higher grade, pathology (adenocarcinoma vs squamous cell carcinoma), location of tumor invasion, TNM stage, lymph biopsy, surgery, regional lymph nodes examination, distant site metastases (liver, brain, and lung), and tumor size (≥5cm) contributed to higher PM incidence. In 884 external data sets, 19 (2.15%) patients were diagnosed with PM. The clinicopathological characteristics of patients with or without pulmonary metastasis were presented in Table 1.

Table 1

Baseline Demographic and Clinical Characteristics of Included Patients Diagnosed with and without PM

Variables	Level	Total (N=22766)	Yes (N=998)	No (N=21768)	P-value
Age, y (median [IQR])		49.00 [39.00, 61.00]	58.00 [49.00, 68.]	49.00 [39.00, 61.00]	<0.001
Race (%)	White	16,957 (74.5)	731 (73.2)	16,226 (74.5)	<0.001
	Black	3153 (13.8)	172 (17.2)	2981 (13.7)
	Other	2403 (10.6)	95 (9.5)	2308 (10.6)
	Unknown	253 (1.1)	0 (0.0)	253 (1.2)
Year (%)	2010	3220 (14.1)	113 (11.3)	3107 (14.3)	0.038
	2011	3193 (14.0)	126 (12.6)	3067 (14.1)
	2012	3242 (14.2)	152 (15.2)	3090 (14.2)
	2013	3079 (13.5)	130 (13.0)	2949 (13.5)
	2014	3309 (14.5)	150 (15.0)	3159 (14.5)
	2015	3364 (14.8)	173 (17.3)	3191 (14.7)
	2016	3359 (14.8)	154 (15.4)	3205 (14.7)
Site* (%)	Cervix uteri	17,682 (77.7)	870 (87.2)	16,812 (77.2)	<0.001
	Endocervix	4321 (19.0)	108 (10.8)	4213 (19.4)
	Exocervix	411 (1.8)	10 (1.0)	401 (1.8)
	OLC	352 (1.5)	10 (1.0)	342 (1.6)
Grade (%)	Grade I	2536 (11.1)	18 (1.8)	2518 (11.6)	<0.001
	Grade II	7029 (30.9)	172 (17.2)	6857 (31.5)
	Grade III	6427 (28.2)	415 (41.6)	6012 (27.6)
	Grade IV	542 (2.4)	46 (4.6)	496 (2.3)
	Unknown	6232 (27.4)	347 (34.8)	5885 (27.0)
Pathology (%)	ADC	4276 (18.8)	141 (14.1)	4135 (19.0)	<0.001
	SCC	14552 (63.9)	573 (57.4)	13,979 (64.2)
	Others	3938 (17.3)	284 (28.5)	3654 (16.8)
SEER sit£ (%)	Localized	8788 (38.6)	0 (0.0)	8788 (40.4)	<0.001
	Regional	7421 (32.6)	0 (0.0)	7421 (34.1)
	Distant	2822 (12.4)	844 (84.6)	1978 (9.1)
	Unknown	3735 (16.4)	154 (15.4)	3581 (16.5)
T stage₰ (%)	T0	13 (0.1)	2 (0.2)	11 (0.1)	<0.001
	T1	10,462 (46.0)	90 (9.0)	10,372 (47.6)
	T2	4242 (18.6)	143 (14.3)	4099 (18.8)
	T3	3045 (13.4)	323 (32.4)	2722 (12.5)
	T4	724 (3.2)	112 (11.2)	612 (2.8)
	TX	767 (3.4)	159 (15.9)	608 (2.8)
	Unknown	3513 (15.4)	169 (16.9)	3344 (15.4)
N stage (%)	N0	13,676 (60.1)	223 (22.3)	13,453 (61.8)	<0.001
	N1	4834 (21.2)	472 (47.3)	4362 (20.0)
	NX	743 (3.3)	134 (13.4)	609 (2.8)
	Unknown	3513 (15.4)	169 (16.9)	3344 (15.4)
M stage (%)	M0	16,632 (73.1)	0 (0.0)	16,632 (76.4)	<0.001
	M1	2621 (11.5)	829 (83.1)	1792 (8.2)
	Unknown	3513 (15.4)	169 (16.9)	3344 (15.4)
Lym biopsy (%)	<3	466 (2.0)	9 (0.9)	457 (2.1)	<0.001
	≥4	7264 (31.9)	27 (2.7)	7237 (33.2)
	Unknown	15,036 (66.0)	962 (96.4)	14,074 (64.7)
Surgery (%)	Yes	12,582 (55.3)	85 (8.5)	12,497 (57.4)	<0.001
	No	10,030 (44.1)	909 (91.1)	9121 (41.9)
	Unknown	154 (0.7)	4 (0.4)	150 (0.7)
Lym examination (%)	Negative	14,281 (62.7)	909 (91.1)	13,372 (61.4)	<0.001
	Positive	8161 (35.8)	60 (6.0)	8101 (37.2)
	Unknown	324 (1.4)	29 (2.9)	295 (1.4)
Bone met (%)	Yes	537 (2.4)	223 (22.3)	314 (1.4)	<0.001
	No	22,185 (97.4)	750 (75.2)	21,435 (98.5)
	Unknown	44 (0.2)	25 (2.5)	19 (0.1)
Brain met (%)	Yes	85 (0.4)	48 (4.8)	37 (0.2)	<0.001
	No	22,640 (99.4)	925 (92.7)	21,715 (99.8)
	Unknown	41 (0.2)	25 (2.5)	16 (0.1)
Liver met (%)	Yes	486 (2.1)	244 (24.4)	242 (1.1)	<0.001
	No	22,245 (97.7)	734 (73.5)	21,511 (98.8)
	Unknown	35 (0.2)	20 (2.0)	15 (0.1)
Lym distant (%)	Yes	269 (1.2)	73 (7.3)	196 (0.9)	<0.001
	No	3076 (13.5)	77 (7.7)	2999 (13.8)
	Unknown	19,421 (85.3)	848 (85.0)	18,573 (85.3)
Tumor size, cm (%)	<5	13,256 (58.2)	426 (42.7)	12,830 (58.9)	<0.001
	≥5	455 (2.0)	1 (0.1)	454 (2.1)
	Unknown	9055 (39.8)	571 (57.2)	8484 (39.0)
Insurance (%)	Medicaid	6881 (30.2)	355 (35.6)	6526 (30.0)	0.001
	Insured	13,794 (60.6)	553 (55.4)	13,241 (60.8)
	Uninsured	1412 (6.2)	65 (6.5)	1347 (6.2)
	Unknown	679 (3.0)	25 (2.5)	654 (3.0)
Marital status (%)	Married	14,345 (63.0)	641 (64.2)	13,704 (63.0)	0.217
	Unmarried	6984 (30.7)	307 (30.8)	6677 (30.7)
	Unknown	1437 (6.3)	50 (5.0)	1387 (6.4)

Notes: *According to the primary site labeled. ₰According to the American Joint Committee on Cancer(AJCC), 6th. £According to the SEER historic stage (1973–2015).

Abbreviations: IQR, interquartile range; OLC, overlapping lesion of cervix uteri (cervix uteri, endocervix, exocervix equivalent FIGO I, overlapping lesion of cervix uteri equivalent FIGO II); ADC, adenocarcinoma; SCC, squamous cell carcinoma; Lym biopsy, lymph node biopsy; Lym exam, lymph node examination; Bone met, bone metastasis; Brain met, brain metastasis; Liver met, liver metastasis; Lym distant, distant lymphatic metastasis.

Assessment of Risk Factors for PM in Patients with Cervical Cancer

The GLM, as a traditional linear regression model, was first developed for multivariable analysis, dealing with both category and continuous variables. The best model has then to be identified based on the Akaike Information Criterion (AIC). Candidate variables (AIC=5982.0) such as age at diagnosis, tumor size, grade, T stage, bone metastasis, brain metastasis, and liver metastasis were significantly associated with PM ().

These independently associated risk factors were used to form a PM risk estimation nomogram (Figure 2A). The resulting model was internally validated using the bootstrap validation method. The AUC (0.81, 95% CI:0.79–0.82) Brier score (3.5, 95% CI:3.3–3.7) measures the accuracy of probabilistic predictions (Figure 2B). The predicted accuracy of the nomogram was also validated via the risk histogram (Figure 2C). Besides, the random allocation was used to assess the RFM, the importance of variables could be reflected by the mean decreased Gini (MDG) index. As indicated in Figure 3A, sixteen variables were ordered according to the MDG index. Consistent with the variables screened by GLM, TNM stage, pathology, tumor size, liver metastasis, bone metastasis, brain metastasis, and distant lymph metastasis was more significant for PM prediction (). The relationship of variables changes between the prediction error and the number of decision trees were shown in Figure 3B. A total of 6830 testing samples were adopted to assess the efficacy of the prediction model, and the results revealed that RFM performed better than GLM (Figure 3C).

Figure 2

Generalized linear model. (A) Nomograms conveying the results of the candidate factors for predicting micrometastasis of lymph nodes. (B) Calibration curves for internal validation of the nomogram. (C) Predicted risk histogram comparing predicted risk of the nomogram with the observed frequency.

Figure 3

Random forest model. (A) The candidate factors associated with micrometastasis of lymph nodes were ordered according to the mean decreased Gini index. (B) Relationship of dynamic changes between the prediction error and the number of decision trees. (C) Performance of the prediction model with increasing numbers of features in the ROC curve.

Comparison Between Machine Learning-Based Models

The predictive performance of the five models was shown in . The best performance was observed in the RFM (AUC =0.97, 95% CI:0.96–0.99), which performed similarly to DT (AUC =0.96, 95% CI:0.95–0.96), ANN (AUC=0.94, 95% CI:0.82–0.99), and NBM (AUC=0.92, 95% CI:0.91–0.93). Moreover, all models performed significantly better than conventional methods (GLM). Consistent with the results of training set, the external data set also confirmed that the RFM model was optimal (AUC =0.95, 95% CI:0.46–1.44).

Survival Outcomes and Prognostic Factors for Cervical Cancer Patients with PM

The CPHM and CRM were also included for survival outcome prediction. The multivariate Cox analysis showed that age at diagnosis, surgery, race, bone metastasis, brain metastasis, liver metastasis, and distant lymph metastasis significantly correlated with OS (). Similarly, the age at diagnosis, surgery, bone metastasis, brain metastasis, liver metastasis, and distant lymph metastasis were also correlated with CSS (). Based on this, independent prognostic factors were used to establish the OS and CSS nomograms (Figures 4 and 5). The C-index of the OS nomogram was 0.69 and the concordance index of the CSS nomogram was 0.63. In addition, validation showed consistent agreement between the risk of survival outcome and the actual estimation by the testing cohort.

Figure 4

Cox proportional hazard model. (A) Nomograms for 1-, 3-, and 5-year overall survival (OS) prediction. (B) Calibration curves for internal validation of the nomogram. (C) Predicted risk histogram comparing predicted risk of the nomogram with the observed frequency.

Figure 5

Competitive risk model. (A) Nomogram predicting CSS at 1-, 3- and 5-year using the competitive risk model. According to the Nomogram score, patient No.31 has a cumulative risk of 0.675, 0.882, and 0.928 at 1-, 3- and 5-year, respectively. (B) Calibration curves for internal validation of the nomogram. (C) Predicted risk histogram comparing predicted risk of the nomogram with the observed frequency. *P<0.05; ***P<0.001.

Discussion

For patients with cervical cancer, 4.16% to 7.7% of patients develop PM.1,22 Although PM from primary cervical carcinoma is rare, the number of metastatic nodules, pulmonary metastasectomy, and the disease-free interval between the primary gynecologic procedure is tightly associated with survival outcome.14,23 Nowadays, preclinical studies have shown that PD-1 blockade can inhibit tumor growth and even reduce metastasis, which may bring a glimmer of hope for the prognosis of PM in patients with cervical cancer.24

Owing to this lack of a specific and practical predictive method, the development of a predictive model that incorporates factors associated with PM based on clinicopathological characteristics data becomes desirable. This population-based study explored the relationship between cervical cancer patients with PM, survival outcomes, and risk factors, which is essential for designing effective treatment strategies. To our knowledge, this study is the first to investigate the risk factors and prognostic factors in cervical cancer patients associated with PM based on machine learning algorithms.

In this study, using machine learning algorithms, we developed five models to predict PM in cervical cancer patients, the ROC analysis was used to evaluate the predictive performance and clinical values of these models, respectively. Based on the supervised learning method, most predictive models maintained high AUCs (ranging from 0.919 to 0.972), all of these models performed better than conventional logistic regression analysis (AUC was 0.805) in the prediction of PM. Thus, reliable predictive models for PM in cervical cancer patients were constructed via machine learning, an accurate estimation of PM presence can help surgeons choose appropriate treatment procedures based on risk-benefit assessment.

Previous studies revealed that senior age, non-squamous type, advanced stage, lymph nodes metastases, and poor differentiation were associated with an increased risk for PM.1,14,23,25 In this study, other factors that have been recognized to be important for PM formation were not included in the predictive model, such as organ-site metastasis, TNM stage. Of the currently available prediction tools, the predictive performance indicated that the random forest algorithm was most effective than other models in distinguishing the possibility of PM. Unlike other established risk assessment methods, the random forest algorithm can achieve higher accuracy in the disease prediction by using bootstrap aggregation and randomization of predictors.26,27 Interestingly, using machine learning algorithms, we found that the age, tumor size, histological grade, organ-site metastasis, pathologic types, and TNM stage were indicated to be the most contributive risk factors of PM, consistent with the results of linear regression analysis. However, in this study, the aforementioned factors were included, and correlations within those clinicopathological characteristics were considered, which accounted for a better result than multivariable logistic regression.

Until now, the Cox proportional hazard model has been the most widely applied to predict survival outcomes. Due to the rarity of PM in cervical cancer patients, limited studies as to prognosis have been launched before.12,28,29 In the present study, we identified several factors, including older age, surgery, race, organ site metastasis (bone, brain, liver), and distant lymph metastasis, were identified as prognostic factors for cervical cancer patients with PM at diagnosis. The median survival after PM was eighteen months with 2- and 5-year survival rates of 37.7% and 7.5%, respectively.1 Traditionally, several widely recognized staging systems are available for evaluating the prognosis of cervical cancer patients, such as the Federation International of Gynecology and Obstetrics (FIGO) staging, TNM staging systems, and American Joint Committee (AJCC) cancer staging system. However, these staging systems and prognostic factors were only applied to cervical cancer patients without PM. In this study, we developed the proposed nomogram, which incorporated seven comprehensive and easily available variables, performed well as supported by the C index values in the training and validation cohorts, respectively. In the PM risk estimation nomogram, age, surgery, organ site metastasis, and distant lymph metastasis have been reported to decrease the possibility of better prognosis in advanced cervical cancer.1,30–32 Besides, when cervical malignancy metastasizes to the pulmonary, surgery is a good choice because it provides a survival advantage.33,34 Collectively, based on these survival predictions, the nomogram might serve as a tool to select patients for evaluating the efficacy of pulmonary resection in patients with early cervical cancer and different risks of PM.

This large-scale study revealed the clinical characteristics, risk, and prognostic factors for cervical cancer patients with PM. However, there were also some limitations in our study. First, this analysis was based on data from the SEER database, as mentioned above, information on active surgical resection of the pulmonary lesion(s), and non-operative therapy could not be obtained, which was recommended to improve the prognosis; Second, although the nomogram achieved definite predictive accuracy of PM, however, the estimation of prognosis for the cervical cancer patient with PM remained relative inefficiency, extra laboratory parameters, which might play crucial roles in determining the prognosis; Third, this was a retrospective study that could not completely avoid data selection and measurement biases, it is necessary to validate the results from more prospective studies or multicenter studies in the future.

Conclusion

The deep learning signature could be used for risk stratification in cervical cancer patients with or without PM, and the random forest model provided an optimal estimation of PM risk in cervical cancer patients. The feature selection approach identified that age, surgery, advanced stage, organ site metastasis, lymph nodes metastases, and poor differentiation were associated with an increased risk for PM, some of which also correlated with the survival outcome.