Development and validation a radiomics nomogram for diagnosing occult brain metastases in patients with stage IV lung adenocarcinoma.

BACKGROUND: To develop and validate a radiomics model using computed tomography (CT) images acquired from the first diagnosis to estimate the status of occult brain metastases (BM) in patients with stage IV lung adenocarcinoma (LADC).
METHODS: One hundred and ninety-three patients who were first diagnosed with stage IV LADC were enrolled and divided into a training cohort (n=135) and a validation cohort (n=58). Then, 725 radiomic features were extracted from contoured primary tumor volumes of LADCs. Intra- and interobserver reliabilities were calculated, and the least absolute shrinkage and selection operator (LASSO) was applied for feature selection. Subsequently, a radiomics signature (Rad-Score) was built. To improve performance, a nomogram incorporating a radiomics signature and an independent clinical predictor was developed. Finally, the established signature and nomogram were assessed using receiver operating characteristic (ROC) curves and precision-recall curves (PRC). Both empirical and α-binomial model-based ROCs and PRCs were plotted, and the area under the curve (AUC) and average precision (AP) of ROCs and PRCs were calculated and compared.
RESULTS: A radiomics signature and Rad-Score were constructed using eight radiomic features, and these had significant correlations with occult BM status. A nomogram was developed by incorporating a Rad-Score and the primary tumor location. The nomogram yielded an optimal AUC of 0.911 [95% confidence interval (CI): 0.903-0.919] and an AP of 0.885 (95% CI: 0.876-0.894) in the training cohort, and an AUC of 0.873 (95% CI: 0.866-0.80) and an AP of 0.827 (95% CI: 0.820-0.834) in the validation cohort using α-binomial model-based method. The calibration curve demonstrated that the nomogram showed high agreement between the actual occult BM probability and predicted by the nomogram (P=0.427).
CONCLUSIONS: The nomogram incorporating a radiomics signature and a clinical risk factor achieved optimal performance after holistic assessment using unbiased indexes for diagnosing occult BM of patients who were first diagnosed with stage IV LADC. 2021 Translational Cancer Research. All rights reserved.

Introduction

Lung cancer is the most common cause of cancer-related deaths worldwide, of which 80–85% are non-small cell lung cancer (NSCLC) (1,2). Brain metastases (BMs) remain an important cause of morbidity and mortality in patients with lung cancer, especially NSCLC (3).

Lung adenocarcinoma (LADC) is the most common histologic subtype of NSCLC and is the dominant epidemic factor of BM, and it usually leads to death (4). Therefore, the prevention of BM is of great significance for patients with LADC. However, approximately 10% of NSCLC patients already have BM at first diagnosis (5). Notably, the rate of BM in advanced-stage patients with NSCLC who also have bone metastases, liver metastases, or/and adrenal metastases is higher than that in patients at earlier stages (6). Moreover, LADCs with distant bone metastases and/or liver metastases who were diagnosed with stage IV disease are more likely to develop BM than those diagnosed at earlier stages (7). In this study, “occult” refers to cases where BMs are not found using computed tomography (CT) but are found when using magnetic resonance imaging (MRI). Treatment strategies are distinctly different depending on whether occult BM is present, and hence early detection of occult BM and delivery of tailored therapies, including radiotherapy (8), surgery (9), or/and tyrosine kinase inhibitors (TKIs) (10,11), can significantly improve the prognosis of stage IV LADC.

Occult BM usually measuring less than 1 cm may be ignored during scanning CT. Guidelines for NSCLC by the National Comprehensive Cancer Network (NCCN) recommend MRI to detect single or multiple intracranial BM (12). However, although cranial MRI is considered the gold standard for diagnosing occult BM, it is not suitable for patients with heart pacemakers, claustrophobia, metal implants, or low Karnofsky Performance Status (KPS) (13-15). Advances in computational image analysis, such as radiomics, involve the use of large amounts of quantitative imaging features derived from medical images to decode tumor pathology or heterogeneity (16-18). Recently, many studies aimed to construct CT radiomic models to assess the relationship between lung cancer and BM (19,20). Zhang et al. extracted radiomics features from contrast-enhanced brain CT to differentiate the pathological subtypes of primary lung cancer (21).

On the other hand, Chen et al. built a hybrid model incorporating clinical risk factors and unenhanced CT radiomics features to predict BM in patients with T1 LADCs (22). In contrast, a radiomics model was developed by Sun et al., which can accurately assess the BM-free survival of patients with locally advanced LADC following thoracic surgery (9). However, to the best of our knowledge, no previous radiomic studies have been conducted to address the diagnosis of occult BM in stage IV LADC, even though it has potential clinical applications that can improve prognosis. Therefore, a reasonable hypothesis is that CT radiomics can be utilized to predict occult BM in LADCs as a feasible alternative to MRI scanning, thus lessening the burden on patients and the health care system.

In the present study, we aimed to develop and validate a radiomics model using CT images acquired at first diagnosis to estimate the status of occult BM in patients with stage IV LADCs. We present the following article in accordance with the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) reporting checklist (available at https://dx.doi.org/10.21037/tcr-21-702).

Methods

Study design

The overall workflow of this study is illustrated in . The workflow consists of five steps: (I) patients and images, (II) tumor segmentation, (III) feature extraction, (IV) radiomics signature and nomogram construction, and (V) model evaluation.

Figure 1

Workflow of study.

Patient sample

The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). This study was approved by the Institutional Review Board of Shandong University (KYLL-2014(LW)03), and the need for informed consent was waived because the study was an observational, retrospective study. In this study, patients who were first diagnosed with stage IV LADC between January 1, 2014, and July 31, 2020, and who had distant metastases (brain, lung, bone, liver, adrenal gland, or other organs) were included in the study. The inclusion criteria were as follows: (I) clinical diagnosis with stage IV lung cancer, (II) no BM observed in pretreatment baseline cranial CT images, and (III) had cranial MRI and baseline chest CT. The exclusion criteria were as follows: (I) no pretreatment baseline chest CT images, (II) insufficient image quality, (III) no cranial MRI scans, and (IV) a history of other malignancy. The occult BM status of enrolled patients was determined using CT and MRI, which were cross-reviewed by two experienced radiologists. Finally, enrolled patients were divided into two cohorts using a cutoff date of July 1, 2018, i.e., the training cohort to develop the prediction model and the validation cohort to evaluate the performance of the established model.

Clinical factors and laboratory indexes, including age, sex, smoking status, tumor diameter, CT-reported lymph node (LN), primary tumor location, pleural effusion, and carcinoma embryonic antigen (CEA), were recorded.

CT images acquisition

CT images were scanned and collected for all enrolled patients. All patients underwent standard chest unenhanced CT scanning using a GE scanner (Light Speed VCT, GE Healthcare Systems, Chicago, IL, USA) or a GE CT scanner (Discovery 750 HD, GE Healthcare Systems, Chicago, IL, USA). The scanning protocol was as follows: X-ray tube peak voltage 120 kV, tube current ranged from 230 to 327 mA, slice thickness 5.0 mm, in-plane resolution 0.7734×0.7734 mm/pixel, and helical scanning mode.

Tumor segmentation

Tumor segmentation was implemented using 3D Slicer software (version 4.10.2, www.slicer.org), an open-source platform for medical image processing, including segmentation, registration, visualization, etc. Primary gross tumor volumes (GTV) of LADCs were manually delineated by an independent radiologist with over five years of professional experience, and then another experienced oncologist crosschecked contoured GTVs. Finally, given that contoured GTVs may not be accurate, a 3×3 Gaussian filter with a variance of σ2=0.2 was used to thicken and smoothen the contours to facilitate all delineations more robustly and reduce the risk of interobserver variance.

Radiomic feature extraction

Radiomic features were extracted using the SlicerRadiomics extension module in 3D slicer (version 4.10.2, www.slicer.org), which is an embedded graphical user interface (GUI) of pyRadiomics packages used for specific feature calculations (23) for each contoured GTV. In all, 725 features were calculated, and they fell into the following six categories: (I) 14 shape features, (II) 18 first-order intensity histogram (IH)-based features, and 61 statistical matrix (SM)-based features divided into (III) 24 gray-level co-occurrence matrix (GLCM)-based features, (IV) 16 gray-level run-length matrix (GLRLM)-based features, (V) 16 gray-level size zone matrix (GLSZM)-based features, and (VI) 5 neighboring gray-tone difference matrixes (NGTDM). Moreover, 632 wavelet-based features (including IH and SM features) were extracted from eight wavelet decompositions.

Radiomics signature and nomogram construction

A two-step feature selection was implemented before building the radiomics signature. First, intra- and interobserver reliabilities were tested with 30 randomly selected patients using an intraclass correlation coefficient (ICC), where an ICC ≥0.75 was considered as indicating good reliability. The detailed workflow for feature selection and modeling can be found in Figure S1. Then, the least absolute shrinkage and selection operator (LASSO) (with a binary regression model, a five-fold cross-validation method, an “auc” loss measurement, and using normalized data) was performed to determine the most predictive features for diagnosing occult BM and to make subsequent models or signatures more robust and effective. Moreover, predictive clinical risk factors were selected from potential candidates using univariate analyses. Univariate analyses were performed using the training cohort.

After feature selection, the Rad-Score, also known as the radiomics signature or score, was constructed from a linear combination of features and corresponding weights derived from LASSO. Furthermore, a nomogram was established by incorporating the Rad-Score and predictive clinical risk factors to test whether the predictive performance would benefit from clinical information.

Validation of radiomics signature and nomogram

The Rad-Score and nomogram were developed based on the training cohort and validated in both the training and validation cohorts. In this study, discrimination and calibration were calculated to assess the developed models. First, the Rad-Score and nomogram discrimination were evaluated using an empirical receiver operating characteristic curve (ROC) and the area under the curve (AUC). Due to the limited cohort size, empirical ROC curves have the disadvantage of having an uneven appearance and introducing non-ignorable uncertainty, especially when the dataset is small. ROC curves may perform poorly for evaluation despite a superior AUC when the positive and negative data used for building a prediction model are imbalanced. In these cases, the precision-recall curve (PRC) plots the positive prediction value (PPV) against the true positive rate (TPR) across all thresholds and represents a more accurate method to assess established models and the area under PRC defined as average precision (AP). Hence, the α-binomial model-based ROC curve and PRC proposed by Brodersen et al. to plot smooth curves were used to address the above issues (24). Second, the nomogram calibration was assessed regarding its predictive accuracy and the agreement between the actual occult BM probability and that predicted by the nomogram; using a Hosmer-Lemeshow test, a P value ≥0.05 was considered as indicating good agreement.

Statistical analysis

Statistical analyses were performed in R version 3.3.1 (The R Foundation for Statistical Computing). Comparisons and univariate analyses were implemented in R with the stats package. Mann-Whitney U-tests or two-sample t-tests were employed where appropriate. The LASSO was performed in R with the glmnet package. Nomogram and calibration curves were performed in R with the rms package. The α-binomial model-based ROC curves and PRCs were plotted in MATLAB (version 2018a, The MathWorks Inc., Natick, MA, USA) using an open-source code derived from MathWorks Central File Exchange (25). All of the reported statistical significance levels were two-sided. The statistical significance level was set to P<0.05.

Results

Patients’ characteristics

In total, 193 consecutive patients were enrolled. The training cohort and the validation cohort consisted of 135 and 58 patients, respectively. Patients’ characteristics, including clinical factors and laboratory indexes, are summarized in . The demographic characteristics of the two cohorts were similar. No significant differences were observed for any factors with P values ranging from 0.094 to 0.836. Two examples of occult BM are depicted in .

Table 1

Characteristics of patients in the training and validation cohorts

Characteristic	Training Cohort	Validation Cohort	P value
Age (years)			0.162
Mean	62.39	62.17
Range	28–82	39–83
Sex			0.347
Male	70 [52]	38 [66]
Female	65 [48]	20 [34]
Smoke			0.531
Yes	56 [41]	25 [43]
No	79 [59]	33 [57]
Tumor diameter			0.094
≤4 cm	70 [52]	36 [62]
4–8 cm	59 [44]	19 [33]
≥8 cm	6 [4]	3 [5]
Primary tumor location			0.836
RL-Upper	41 [30]	17 [29]
RL-Middle	9 [7]	5 [9]
RL-Lower	34 [25]	7 [12]
LL-Upper	23 [17]	15 [26]
LL-Lower	21 [16]	12 [21]
Right hilar	2 [1]	2 [3]
Left hilar	5 [4]	0 [0]
Pleural effusion			0.117
Yes	69 [51]	27 [47]
No	66 [49]	31 [53]
CT-reported LN			0.461
Yes	99 [73]	45 [78]
No	36 [27]	13 [22]
CEA			0.253
Normal	49 [36]	22 [38]
Abnormal	86 [64]	36 [62]
Occult BM			0.511
Yes	60 [44]	23 [40]
No	75 [56]	35 [60]

All the data except Age in above table are numbers of patients, with percentages in parentheses. No difference was found between training cohort and validation cohort (P=0.094–0.836). RL, right lung; LL, left lung; CT, computed tomography; LN, lymph node; CEA, carcinoembryonic antigen; BM, brain metastasis.

Figure 2

Two examples of occult brain metastases (BM). (A) and (B) are the same slice in computed tomography (CT) and magnetic resonance imaging (MRI) of Example 1, respectively. (C) and (D) are the same slice in CT and MRI of Example 2, respectively. Red arrows indicated that occult brain metastases.

Radiomics signature construction and validation results

After the first step of feature selection, 549 out of 725 extracted features had ICC ≥0.75, indicating high intra- and interobserver reliabilities for the multiple segmentation test, and the results of reproducible features listed in Table S1. Subsequently, eight features with nonzero coefficients were selected from the reliable features using a LASSO binary regression model while tuning with the parameter λ. The process of the tuning phase of LASSO and the feature dimension reduction is illustrated in .

Figure 3

Feature selection using the least absolute shrinkage and selection operator (LASSO) with a binary classification model. (A) The LASSO coefficient profile graph is shown using coefficients against the log(λ) sequence. Vertical lines are shown corresponding to 1 standard error (1-se) and maximum criteria. (B) Tuning parameter λ-based 1-se and maximum criteria. The area under the curve (AUC) relating to λ is shown. The 1-se of maximum criteria (left vertical line) is shown using the right vertical line. In this study, λ with corresponding optimal AUC was chosen (λ=−2.781). As a result, eight radiomic features with nonzero coefficients were selected.

Then the Rad-Scores were established using the following formulas:

The Rad-Score values calculated from the training and validation cohort are shown in . The empirical and α-binormal-based ROC curves and PRCs of the developed Rad-Scores are presented in .

Figure 4

Bar plot of the Rad-Score value for each patient in the training cohort (A) and the validation cohort (B). Patients with Rad-Score values ≥0 were considered as being at high risk for metastases, while patients with values <0 were deemed as being at low risk for metastases. Patients with and without BM are marked with different colors.

Figure 5

The performance of the Rad-Score developed in this study. (A) The receiver operating characteristic (ROC) curves. (B) The precision-recall curve (PRC). Subscripts emp and αbin are empirical and α-binormal-based, respectively, ROC or PRC.

Nomogram construction and validation

In univariate analyses, only primary tumor location was an independent clinical risk factor in the nomogram. The results of univariate analyses can be found in Table S2. A nomogram was constructed by combining the independent clinical risk factor and the Rad-Score; it is depicted in . The empirical and α-binormal-based ROC curves and PRCs of developed Rad-Scores are presented in . The calibration curve of the nomogram is plotted in . The Hosmer-Lemeshow test (P value was 0.427) showed that there was no significant difference between predicted metastases and the observed metastases in the validation cohort.

Figure 6

Nomogram developed by incorporating clinical risk factors and Rad-Score of the training cohort.

Figure 7

The performances of the nomogram developed in this study. (A) Receiver operating characteristic (ROC) curves. (B) Precision-recall curve (PRC). Subscripts emp and αbin are empirical and α-binormal-based, respectively, ROC curves or PRCs.

Figure 8

Calibration curve of nomogram plotted to assess the agreement between nomogram-predicted metastases and observed metastases of the validation cohort.

Comparisons of developed Rad-Score and nomogram

The developed Rad-Score and nomogram performances were calculated and compared in the training cohort and the validation cohort, as shown in . The nomogram yielded an optimal AUC of 0.911 [95% confidence interval (CI), 0.903–0.919] and an AP of 0.885 (95% CI: 0.876–0.894) in the training cohort, and an AUC of 0.873 (95% CI: 0.866–0.880) and an AP of 0.827 (95% CI: 0.820–0.834) in the validation cohort using an α-binomial-based method. Notably, all of the performances of the nomogram were significantly higher than those of the Rad-Score (all P value <0.001). Additionally, no significant differences were observed between the empirical and α-binormal-based methods (all P value >0.05), indicating that prediction results of occult BM are reasonable and robust.

Table 2

Comparison of performances of Rad-Score and nomogram

Groups	Training Cohort					Validation Cohort
	AUCαbin	AUCemp	APαbin	APemp		AUCαbin	AUCemp	APαbin	APemp
Rad-Score	0.854 (0.849–0.859)	0.852 (0.848–0.856)	0.822 (0.820–0.824)	0.808 (0.801–0.815)		0.792 (0.790–0.794)	0.809 (0.803–0.812)	0.730 (0.718–0.742)	0.689 (0.681–0.697)
Nomogram	0.911 (0.903–0.919)	0.909 (0.906–0.912)	0.885 (0.876–0.894)	0.869 (0.865–0.873)		0.873 (0.866–0.880)	0.866 (0.861–0.871)	0.827 (0.820–0.834)	0.807 (0.796–0.818)
P value	0.001ζ	0.001ε	0.001η	0.001θ		0.001ψ	0.001σ	0.001ρ	0.001μ

Note: All the data in parentheses are 95% confidence interval (CI). Subscript emp and αbin were empirical-based and α-binormal-based, respectively, AUC or AP. ζ, ψ The comparison of AUCαbin, ε, σ The comparison of AUCemp, η, ρ The comparison of APαbin, and θ, μ The comparison of APemp between Rad-Score and nomogram in both training and validation cohort.

Discussion

Patients who already have bone metastases and/or liver metastases when they are first diagnosed are defined as having stage IV LADC (26). Although they may not yet have BM, these patients are at higher risk for BM than those at earlier stages. Notably, some stage IV LADCs may have developed occult BM, which cannot be captured using CT imaging, with the result that occult BM continues to progress and lead to a poor prognosis. Patients harboring occult BM can be detected using an MR scanner; however, several contraindications limit its availability. With the emergence of quantitative imaging analysis in precision medicine, CT radiomics has been widely used for diagnosis. In this study, although Rad-Score displayed a significant correlation with occult BM, we assumed that it would achieve a better performance if combined with other clinical predictors. Indeed our results verified this hypothesis. The proposed CT radiomics-based nomogram for the diagnosis occult BM showed remarkable AUC and AP in both the training and the validation cohort compared to radiomics alone. The results demonstrated that CT is useful for detecting occult BM of stage IV LADC in the absence of MR imaging.

To make the model developed more predictive and robust, feature selection is a crucial step of a radiomics study (27). The Rad-Score consisted of eight features, and nearly all of the selected predictive features were wavelet, similar to the results of several published studies (28-30). A reasonable hypothesis is that the multifrequency decomposition of the original CT image captured underlies information about tumor heterogeneity between BM and BM-free LADCs, which clinicians cannot assess with unassisted vision. In addition, a bar plot showed that a Rad-Score value of zero is a potential diagnostic threshold and achieved satisfactory performance. Therefore, the established Rad-Score can efficiently stratify patients of unknown BM status into high- and low-risk subgroups concerning BM.

Previous studies have suggested that the radiomics signature combined with clinical risk factors can significantly improve the performance of various clinical tasks related to lung cancer (31-34). For instance, a combined model incorporating CT-based radiomics and clinical features outperforms CT-based radiomics alone for predicting distant metastases of NSCLC (35), estimating BM-free survival of curatively resected locally advanced NSCLC (9), and predicting survival after whole brain radiotherapy of NSCLC patients with BM (8). In this study, the results of univariate analyses of clinical risk factors showed that the primary tumor location had significant correlations with occult BM status and that the upper lobe, including the upper right and left lobe, were the most frequently occurring primary metastatic sites for BM, occurring at a rate of 47%. A recent study using multivariate logistic regressions demonstrated that most BM relating to stage IV NSCLCs were derived from the upper lobes (36). Another study reported that primary tumors in the upper lobe of LADCs were more prone to lymphovascular invasion with the highest incidence of 35.73% and mediastinal lymph node metastases (37). Our results are consistent with the studies mentioned above. Hence, the nomogram developed incorporated the primary tumor location and Rad-Score are achieved optimal performance.

On the other hand, to evaluate the developed models using a robust and unbiased approach, α-binomial model-based ROC curves and PRCs were employed and compared to empirical curves. Our results demonstrated that no significant differences were observed between α-binomial model-based and empirical ROC curves and PRCs. Performance measures based on PRCs are helpful to supplement and validate ROC curves. Furthermore, based on a distributional assumption about the underlying decisional values and an explicit estimation of the class-mixture parameter α-binomial model-based curves, the practical limitations raised by empirical approaches to estimate curves can be addressed effectively (24). Therefore, the nomogram is robust and accurate for predicting occult BM.

Our study has several limitations. First, contrast-enhanced CT images were unavailable for many enrolled patients due to the retrospective design of this study. We believe that the results will benefit greatly from contrast information. Second, survival analysis of patients after target treatment was not conducted in this study due to the limited cohort size and follow-up, including BM-free survival, progress-free survival, and overall survival. We will be conducting a future study, which will focus on predicting lung cancer outcomes, including BM occurrence.

Conclusions

We developed and validated a radiomics-based nomogram model that incorporates a radiomics signature derived from unenhanced CT imaging and clinical risk factors to diagnose occult BM in patients who received a diagnosis of stage IV LADC as their first diagnosis. The nomogram may serve as a powerful and robust tool for diagnosing occult BM.