Robust, independent and relevant prognostic 18F-fluorodeoxyglucose positron emission tomography radiomics features in non-small cell lung cancer: Are there any?

In locally advanced lung cancer, established baseline clinical variables show limited prognostic accuracy and 18F-fluorodeoxyglucose positron emission tomography (FDG PET) radiomics features may increase accuracy for optimal treatment selection. Their robustness and added value relative to current clinical factors are unknown. Hence, we identify robust and independent PET radiomics features that may have complementary value in predicting survival endpoints. A 4D PET dataset (n = 70) was used for assessing the repeatability (Bland-Altman analysis) and independence of PET radiomics features (Spearman rank: |ρ|<0.5). Two 3D PET datasets combined (n = 252) were used for training and validation of an elastic net regularized generalized logistic regression model (GLM) based on a selection of clinical and robust independent PET radiomics features (GLMall). The fitted model performance was externally validated (n = 40). The performance of GLMall (measured with area under the receiver operating characteristic curve, AUC) was highest in predicting 2-year overall survival (0.66±0.07). No significant improvement was observed for GLMall compared to a model containing only PET radiomics features or only clinical variables for any clinical endpoint. External validation of GLMall led to AUC values no higher than 0.55 for any clinical endpoint. In this study, robust independent FDG PET radiomics features did not have complementary value in predicting survival endpoints in lung cancer patients. Improving risk stratification and clinical decision making based on clinical variables and PET radiomics features has still been proven difficult in locally advanced lung cancer patients.

Introduction

Despite the emergence of new technologies and treatment options such as tyrosine kinase inhibitors targeted towards mutations, and immune checkpoint inhibitors, the global survival of lung cancer patients has improved only gradually in the last decades [1-4]. Locally advanced non-small cell lung cancer (NSCLC) is a highly heterogeneous disease where only modest improvements in survival have been observed, with the exception of chemoradiotherapy (CRT) patients treated with the anti-PD-L1 antibody Durvalumab whose overall and progression-free survival significantly improved compared to those receiving CRT alone [5]. New approaches are urgently needed for the selection of treatment strategies for NSCLC patients, which are currently determined mainly by TNM staging [6, 7]. In addition to TNM staging, other well-established, reproducible, independent prognostic factors are used to guide clinicians in making treatment decisions, such as Eastern Cooperative Oncology Group (ECOG) performance status [8, 9], weight loss [10], and gender [11]. Numerous other biomarkers have been investigated, although less reproducible, such as histology [12], age [13], serum blood levels [14, 15], mutation status [16], and protein expression levels [17, 18]. In locally advanced NSCLC, treatment selection based on TNM staging and other clinical variables may not be accurate enough for survival probability prediction [19, 20]. Therefore, the search for more accurate reproducible independent prognostic factors is warranted in the context of personalized medicine.

A current field of interest is the assessment of quantitative image features and its complementary value to well-established clinical prognostic models. Radiomics has been introduced as a sophisticated way to extract and mine a large number of quantitative image features, primarily using anatomical CT information [21]. The basic assumption of radiomics is that underlying tumour biology could be captured [22]. This information may actually be better characterized with functional imaging such as 18F-fluorodeoxyglucose Positron Emission Tomography (FDG PET), the gold standard in NSCLC diagnosis and staging, which is able to characterize molecular heterogeneity in lung cancer [23, 24]. It is therefore worthwhile to investigate the prognostic performance of radiomics features from functional imaging such as PET.

Basic PET radiomics features have provided clinically relevant prognostic information for NSCLC patients. Examples include standardized uptake value (SUV) based metrics like maximum, peak, and mean SUV (SUVmax, SUVpeak, and SUVmean, respectively), metabolic tumour volume (MTV), and total lesion glycolysis (TLG) [25-32]. The more advanced PET texture features employed for quantification of tumour heterogeneity, have also been reported to be of prognostic value [33-41]. However, the variable nature of PET imaging makes it difficult to reproduce these results [42, 43].

Furthermore, PET texture features can also be subject to differences in reconstruction settings and delineation methods [44], SUV binning methods [45, 46], and feature calculation methods [47]. It is not yet clear which PET radiomics features are insensitive to all of these factors, and also to what degree.

Regardless of the issues with variability, complementary PET radiomics features should be independent from well-known prognostic SUV metrics, such as MTV and SUVmax. Some investigators reported specific PET texture features that were associated with MTV [37, 39, 47, 48, 49]. In these cases, prognostic texture features would rather act as a surrogate than as an independent variable. Such an association is also not warranted for clinical variables. Hence, the relationship of PET texture features with well-known prognostic factors has to be thoroughly studied too.

With all the confounding factors described above, in combination with the high number of possible radiomics features, it is not surprising that false discovery rates are high amongst FDG PET and CT studies on texture features [50]. Without proven, robust, and independent prognostic PET texture features, it will be challenging to move further in the field. Therefore, this study aims to investigate the repeatability of PET radiomics features, and also assesses the relationship with well-known prognostic factors in PET, such as MTV and SUVmax. The rationale is to identify a group of radiomics features derived from pre-treatment PET imaging that are robust, independent, and prognostic, with possible additional value to current clinical prognostic variables.

Materials and methods

Patient data

Three NSCLC patient cohorts from the Netherlands Cancer Institute (NKI) and one from the Peter MacCallum Cancer Centre (PMCC) were included in this study to develop and validate a radiomics signature. Peter MacCallum Cancer Centre Ethics and Clinical Research Committees approval was granted and all research was performed in accordance with relevant guidelines/regulations. Patient’s written, informed consent was obtained. An overview of the datasets is given in Table 1. Patients were excluded if the primary tumour was smaller than 10 cc or if the patient had stage IV NSCLC at baseline. To detect brain metastases at baseline, the NKI patients were scanned with MR imaging and the PMCC performed FLT baseline scans before treatment.

Table 1

Overview of the four patient cohorts used in the study.

Unless otherwise stated, values represent the median with the range in parentheses. MTV2.5 = metabolic tumour volume obtained using a SUV threshold of 2.5, MTV40 = metabolic tumour volume obtained using a threshold of 40% of the maximum intensity, SUVmax = maximum SUV uptake, OS = overall survival, PFS = progression-free survival, LRS = local recurrence-free survival, DMS = distant metastases-free survival. Nos = not otherwise specified.

	4D PET lung	NKI lung 1	NKI lung 2	PMCC lung 1
No. of patients	70	228	24	40
Age (year)	n/a	64 (36–87)	63 (39–82)	68 (53–86)
Gender	n/a
Male		142	13	29
Female		86	11	11
Disease stage	n/a	IA-IIIC	IIB-IIIC	IB-IIIC
IA		1	0	0
IB		4	0	1
IIA		0	0	3
IIB		16	2	5
IIIA		102	7	15
IIIB		82	13	12
IIIC		23	2	4
T stage	n/a
1		6	1	5
2		78	4	24
3		63	7	9
4		81	12	2
N stage	n/a
0		42	4	5
1		20	2	5
2		130	13	17
3		36	5	13
Histology	n/a
Adeno		80	12	16
Squamous cell		83	7	13
Large cell		8	1	5
Nos or other		57	4	6
GTV (cc)	n/a	118 (15–906)	84 (10–351)	49 (12–544)
MTV2.5 (cc)	62 cc (10–545)	72 (10–693)	91 (11–337)	51 (8–478)
MTV40 (cc)	27 cc (4–169)	31 (3–394)	34 (5–289)	31 (4–378)
SUVmax	11.5 (4.3–55.1)	14.6 (5.9–44.8)	15.7 (6.9–28.3)	16.5 (6.3–33.2)
Median follow-up time (months)	n/a	17	22	24
2-year OS	n/a	40%	46%	54%
2-year PFS	n/a	29%	21%	20%
1-year PFS	n/a	50%	54%	35%
1-year LRS	n/a	58%	71%	47%
1-year DMS	n/a	54%	54%	45%

The repeatability and independence of PET radiomics features was assessed using a 4D PET/CT dataset (4D PET lung) consisting of 70 stage III NSCLC patients. No clinical data was collected for these patients. The second cohort (NKI lung 1) contained 228 patients treated with concurrent chemoradiotherapy (CCRT) for stage IA-IIIC NSCLC in the NKI between 2007 and 2011 as described earlier [51]. The third cohort, also from the NKI (NKI lung 2), consisted of 24 patients with stage IIB-IIIC NSCLC treated between 2013 and 2016, similar as NKI lung 1. The fourth cohort was from the PMCC (PMCC lung 1) and involved 40 stage IB-IIIC NSCLC patients treated with CCRT as previously reported [32].

Clinical endpoints for prognostic model

The primary endpoint used for the prognostic model was two-year overall survival (2-year OS). Overall survival was defined as the time between the start of treatment and date of death. In addition, two-year progression-free survival (2-year PFS), one-year PFS (1-year PFS), one-year local recurrence-free survival (1-year LRS), and one-year distant metastases-free survival (1-year DMS) were also studied. Progression was defined as growth of tumour cells in the primary tumour or involved lymph nodes, or metastases to other organs, or death. LRS was defined as progression in the primary tumour and/or involved lymph nodes as assessed on follow-up scans. DMS was described according to the 8th edition of the TNM classification for NSCLC [52] as evaluated on follow-up scans.

Data acquisition and image reconstruction

Patients from the NKI lung 1 and 2 dataset both underwent a whole-body FDG PET/CT using a Gemini TF or Gemini TF Big Bore scanner (Philips Medical Systems, Cleveland, OH). The reconstruction voxel size of the PET data was 4 × 4 × 4 mm3. Patients fasted for at least 8 h to ensure low levels of serum glucose. Patients with a Body Mass Index (BMI)≤28 were intravenously injected with 190 MBq 18F-FDG, or 240 MBq in case of a BMI>28. Patients were scanned 60 minutes after injection of 18F-FDG. The acquisition time of the PET/CT scanner was 2 minutes per bed position.

In the PMCC lung 1 cohort, whole-body FDG PET/CT scans were acquired on a GE STE (GE Medical Systems, Milwaukee, WI) or Biograph (Siemens Medical Solutions, Erlangen, Germany) scanner. The reconstructed voxel size of the PET data was 4.3 × 4.3 × 3.3 mm3 for the GE STE scanner, and 4.1 × 4.1 × 3.0 mm3 for the Siemens Biograph scanner. Patients fasted for more than 6 hours before 18F-FDG scans. Patients were intravenously injected with 4.2 MBq/kg 18F-FDG. Baseline emission scans were initiated 60 minutes after injection. The acquisition time of the PET/CT scanner was 3 minutes per bed position.

For the 4D PET lung dataset, scans were acquired on a Gemini TF scanner (Philips Medical Systems, Cleveland, OH). The reconstruction voxel size of the PET data was 4 × 4 × 4 mm3. The 4D PET/CT data were reconstructed in 10 phases, and the attenuation in each frame of the 4D PET data was corrected with the corresponding 4D CT frame. The acquisition time of the 4D PET was kept the same as that used for 3D PET [52].

Mid-position scans from 4D PET lung dataset for repeatability testing

The 4D PET/CT data were reconstructed in 10 phases, and from these phases two new mid-position scans were derived [53]. The first mid-position scan was created from the even phases (0, 2, 4, 6, and 8) and is named ‘Mid-P even’, and the odd phases (1, 3, 5, 7, and 9) were used to create the second mid-position scan ‘Mid-P odd’. The even and odd number of frames were selected to keep the amount of tumour motion balanced in both scans. Fig 1 gives an overview of the workflow.

Fig 1

Workflow of the PET mid-position scans.

A 4D PET scan was loaded for each patient consisting of 10 frames, where the odd or even number of frames were selected. A 4D deformation vector field (DVF) was applied to these frames to deform them to the mid-position. Lastly, the mean of the 5 deformed frames was calculated to obtain the PET mid-position scan. For comparison, the PET mid-position scan obtained from 10 frames has been included in the image too. Mid-P = PET mid-position scan.

The source of variability was different in these two mid-position scans compared to a test–retest setting, since the biological tumour variability has been eliminated. In this case, the variability was mostly caused by minor differences in noise-levels and tumour motion, hence robust quantitative features should not differ substantially in outcome.

Tumour segmentation

For each patient in the NKI lung 1, NKI lung 2, and PMCC lung 1 cohort, a volume-of-interest (VOI) enveloping the primary tumour was manually drawn by radiation oncologists using information from both PET and CT imaging. From this VOI, the MTV was auto-segmented on the FDG PET scan. Two auto-segmentation methods were applied: a metabolic tumour region delineation that included all voxel intensities above 2.5 (SUV2.5), and a high intensity delineation that included all voxel intensities that were at least 40% of the SUVmax (SUV40). Auto-segmentation was performed with in-house developed software named Match42 (version 1.0.0) using a Python plug-in. The metabolic tumour volume obtained from SUV2.5 and SUV40 were named MTV2.5 and MTV40, respectively. In the 4D PET lung dataset, a VOI was manually drawn around the primary tumour in one PET mid-position scan, and copied to the second PET mid-position scan. The auto-segmentation was performed on both PET mid-position scans independently.

PET radiomics features

The Pyradiomics toolkit was used for radiomics feature extraction [54]. With this toolkit a total of 105 features were available for feature calculations. These were divided into 18 first-order features, 13 shape features (including metabolic tumour volume), and 74 texture features describing the spatial distribution of voxel intensities. The texture features were derived from the gray level co-occurrence matrix (GLCM; 23 features) [55], gray level run-length matrix (GLRLM; 16 features) [56], gray level size-zone matrix (GLSZM; 16 features) [57], gray level dependence matrix (GLDM; 14 features) [58], and neighbourhood gray tone difference matrix (NGTDM; 5 features) [59]. The mathematical definitions of these features were in compliance with feature definitions as described by the Imaging Biomarker Standardization Initiative (IBSI) [60].

SUV discretization and matrix calculation

Before texture features were extracted, pre-processing steps were required in the form of SUV binning and matrix definition. SUV discretization is an intensity-resampling step, before building the texture matrices on which texture features rely. SUV discretization or binning was applied with the fixed bin count method (e.g. 64 bins) and an alternative method using a fixed bin width (e.g. 0.25 SUV). All texture features were calculated from a single matrix taking into account all 13 directions simultaneously. A more detailed description on SUV binning and matrix calculation can be found in S1 File, respectively.

Repeatability

The repeatability assessment was performed within the same patient comparing two different PET mid-position scans. For each patient, the PET mid-position scan obtained from the even numbered frames (Mid-P even) was compared with the PET mid-position scan from the odd numbered frames (Mid-P odd). This resulted in four comparisons: 2 SUV binning methods and 2 thresholding methods were applied.

The repeatability of each PET radiomics feature was assessed with the Coefficient of Repeatability (CR) [61]. See S1 File for more details. The CR was reported as a percentage: , where mean is the average of the PET radiomics feature value within the patient cohort. The threshold for poor repeatability was set to a value of 30%, corresponding to PET Response Criteria in Solid Tumours (PERCIST) [62].

Independence testing

To determine whether the features were correlated with the two commonly reported prognostic PET features MTV and SUVmax, the Spearman’s rank correlation coefficient (ρ) was calculated on one of the Mid-P scans, using the same set-up as for the repeatability testing. PET radiomics features that had a |ρ|≥0.5 were considered to have a correlation with MTV or SUVmax, and were discarded from further analysis. The choice of |ρ|<0.5 as limit for independent features was validated with the ‘elbow method’ using hierarchical clustering [63].

An overview of the radiomics workflow and feature selection procedure is given in Fig 2.

Fig 2

Radiomics feature selection workflow: From PET image segmentation to selected features.

Features from MTV2.5 and MTV40 were seen as a separate set of features, doubling the amount of features in the analysis. This also counts for features calculated with fixed bin width and fixed bin count, except for most intensity and shape features that were not affected by SUV discretization. An exception was observed for first-order features Uniformity and Entropy. A total of 360 PET radiomics features were entered into the analysis, including SUVmax, MTV2.5, and MTV40. PET radiomics features were selected for further analysis when two criteria were met: high repeatability and low association with MTV and SUVmax. SUV2.5 = SUV threshold of 2.5; SUV40 = SUV threshold of 40% of maximum SUV; MTV2.5 = metabolic tumour volume obtained from use of SUV2.5; MTV40 = metabolic tumour volume obtained from use of SUV40. GLCM = gray level co-occurrence matrix; GLRLM = gray level run-length matrix; GLSZM = gray level size-zone matrix; GLDM = gray level dependence matrix; NGTDM = neighbourhood gray tone difference matrix; CR = coefficient of repeatability.

Model training

An elastic net regularized generalized logistic regression model (GLM) was built with PET radiomics features derived from pre-treatment PET imaging (GLMrad). To increase the sample size in the training and test sets, for the purpose of building a GLM, NKI lung 1 and lung 2 were combined. In this study, 80% of the NKI data was used for training the model, and 20% for validation. Different ratios of training/validation were also tested, but were not reported as there was no major differences seen in the results. Elastic net regression analysis using the R package ‘glmnet’ was performed on the training set [64]. With 20-fold cross validation (CV), the most optimal fitted GLMrad with minimal CV error was determined and selected for model validation.

Model validation

To validate the fitted model of the training set, the area under the receiver operating characteristic curve (AUC) was calculated between the predicted outcome and the observed outcome in the validation set. To reduce randomness introduced by selecting a random subset of the complete data for training and validation, the procedure for model training and validation was repeated 100 times. This yields a better estimate of the true validation set performance by randomly simulating many scenarios with varying training and validation set compositions [65]. From the 100-times-repeated training/validation procedure, results were averaged, and the best performing GLMrad was externally validated for each clinical endpoint on PMCC lung 1.

During 100-times-repeated training/validation procedure, per iteration, the fitted model was stored to keep track of the PET radiomics features that were selected by elastic net in the fitted model [66]. PET radiomics features and clinical variables were ranked based on the frequency of inclusion in the fitted model.

Model comparison

Clinical variables such as PET/CT-based GTV, TNM staging, histology, gender, and age were also introduced into the radiomics signature to create a prognostic model containing PET radiomics features and clinical variables (GLMall). In addition, a model based on only the clinical variables was calculated using elastic net regression (GLMclin). To assess the complementary value of PET radiomics features with clinical variables, the mean AUC was calculated from 100 iterations for each model and compared. The Mann Whitney U Test was used to assess any significant differences between the predictive performance of GLMall, GLMclin, and GLMrad, and p-values below 0.05 were seen as significant.

Results

Results of the repeatability test were based on the 4D PET lung dataset and an overview of notable PET radiomics features and their corresponding CR is given in Table 2. All first-order features were repeatable when extracted from MTV2.5 irrespective of SUV binning method. In contrast, 13 out of 18 first-order features were repeatable when extracted from MTV40. Furthermore, around 50 texture features were repeatable when extracted from MTV2.5 regardless of SUV discretization method, versus 28 repeatable texture features extracted from MTV40. With regards to shape features, only MTV40 was not repeatable.

Table 2

An overview of categorized notable PET radiomics features that are commonly reported in literature with their coefficient of repeatability (CR, %).

The asterisk (*) represents features that were repeatable in all four different settings. Per category, the total number of PET radiomics features that met the study repeatability criterion is added.

CR (%)	Fixed bin width		Fixed bin count
Notable features	MTV2.5	MTV40	MTV2.5	MTV40
First-order features	18/18	13/18	18/18	13/18
Entropy*	3.4	5.5	3.8	6.0
Kurtosis	26.8	34.7	26.8	34.8
Skewness	23.1	50.4	23.1	51.3
SUVmax*	13.2	13.2	13.2	13.2
SUVmean*	6.0	12.9	6.0	12.7
Uniformity	17.9	41.9	21.1	37.0
Texture features	49/74	28/74	50/74	28/74
GLCM Contrast*	23.2	28.1	28.8	29.9
GLCM Correlation*	2.6	11.9	2.7	11.2
GLCM DifferenceAverage*	9.9	13.1	14.1	17.2
GLCM JointEntropy*	2.8	4.5	3.3	5.7
GLCM SumEntropy*	2.7	4.5	2.8	4.2
GLRLM GrayLevelNonUniformity	13.7	59.1	18.7	55.4
NGTDM Busyness	75.5	91.3	33.0	81.9
NGTDM Coarseness	12.0	41.5	16.8	35.2
NGTDM Contrast	23.3	68.5	31.9	64.9
Shape features	13/13	12/13	13/13	12/13
Elongation*	4.7	10.8	4.7	10.7
Flatness*	7.1	15.0	7.1	13.7
MetabolicTumourVolume	5.9	45.5	5.9	45.3
Sphericity*	3.2	8.9	3.2	8.7

Amongst the four comparisons, 211 out of 360 PET radiomics features were repeatable. An overview of all PET radiomics features and their corresponding CR is given in S1 File. The impact of large delineation inaccuracies on repeatability was studied between contours generated by the two different SUV thresholds, though only reported as supplementary data (S1 File).

Relationship of PET radiomics features with MTV and SUVmax

The Spearman’s Rank correlation coefficient was calculated to assess the relationship of 211 repeatable PET radiomics features with MTV and SUVmax. Four assessments were performed in total on one of the mid-position scans, with groups consisting of a combination of either one of the SUV binning methods and one of the tumour volumes (MTV2.5 or MTV40). Not all repeatable PET radiomics features were found to be independent from MTV and SUVmax. From the first-order features, only Kurtosis and Skewness extracted from MTV2.5 were independent from MTV and SUVmax. There were no independent repeatable first-order features for MTV40. Regarding the fixed bin count method, 17 out of 50 texture features extracted from MTV2.5 were not strongly associated with MTV and SUVmax. This also counted for 5 texture features extracted from MTV40. With regards to the fixed bin width method, there were no texture features independent from either SUVmax or MTV. Elongation, Flatness, and Sphericity were the only independent shape features when extracted from MTV2.5, though only Elongation and Flatness remained independent for MTV40. A complete overview of independence testing for all PET radiomics features is given in S1 File.

An overview of correlations amongst the selected robust independent PET radiomics features and clinical variables is given in Fig 3. More details on robust and independent PET radiomics features can be viewed in S1 File. The robust independent PET radiomics features did not show any strong correlation with the other clinical variables, such as age, ECOG PS, gender, histology, and TNM stage. However, there were associations present amongst the PET texture features.

Fig 3

Correlation coefficients of the robust independent PET radiomics features and clinical variables.

Positive correlation coefficients are displayed in blue and negative correlation coefficients in red color. Color intensity and the size of the circle are proportional to the correlation coefficients. A distinction was made between features calculated from MTV2.5 and MTV40.

Building the radiomics signature

Based on the feature selection criteria, 31 PET radiomics features were selected for the next steps (see Fig 3). Three elastic net regularized GLMs were built per endpoint: GLMrad, GLMclin, and GLMall. Results of the model performances are shown in Fig 4, showing that GLMrad does not significantly outperform GLMclin for any clinical endpoints. The GLMclin has a significantly better predictive performance compared to GLMrad in 2-year OS (p<0.0001), and in 1-year LRS (p<0.001). GLMall did not show a significantly better performance to both GLMrad and GLMclin simultaneously in any endpoint. External validation of GLMall led to AUC values ranging from 0.51 to 0.59 for any clinical endpoint. When GLMclin was externally validated, the highest predictive performance was 0.60 for 2 year OS. For GLMrad, the highest predictive performance was 0.71 for 2-year PFS.

Fig 4

Model performance for the PET radiomics model (GLMrad), the model containing clinical variables (GLMclin), and a combination of radiomics and clinical variables (GLMall).

The median AUC values from 100-times-repeated training/validation are depicted per model, per clinical endpoint. The lower and upper hinges correspond to the 25th and 75th percentiles. The whiskers depict the 1.5*IQR from the lower and upper hinge. Data beyond the end of the whiskers are shown as outlier points. AUC values corresponding to the external validation set are shown as a black diamond. Significance levels, **p<0.001, ***p<0.0001.

Promising features

Table 3 shows selected features for each fitted GLM, and how frequent these features were chosen in the fitted model over 100 iterations. The feature shape Sphericity was present in 100% of the iterations for 2-year OS. From the 100 repetitions, GLCM ClusterTendency was selected in more than 95% for predicting 1-year PFS and 1-year DMS. Clinical variables such as age and GTV were prominent in predicting 2-year OS and 1-year LRS, next to shape Sphericity. As can be seen in Table 3, age, shape Sphericity, and GLCM ClusterTendency are present amongst the most selected features for all clinical endpoints.

Table 3

The most selected features in the model by elastic net, ranked by the number of times selected in the generalized linear model.

Only the top 10 most selected PET radiomics are shown. The features written in italic bold are present in all endpoints.

Endpoint	GLMall selected features by elastic net	Frequency
2-year OS	Age	100
	GTV	100
	Shape_Sphericity	100
	MTV2.5	78
	glcm_ClusterTendency	56
	SUVmax	39
	Gender	34
	glcm_JointEntropy	34
	glrlm_GrayLevelNonUniformityNormalized	33
	glrlm_GrayLevelVariance	29
2-year PFS	Age	50
	SUVmax	50
	glrlm_GrayLevelNonUniformityNormalized	49
	shape_Sphericity	47
	Histology	42
	MTV2.5	38
	glcm_ClusterTendency	30
	N_status	28
	T_status	25
	shape_Elongation_MTV40	20
1-year PFS	GTV	99
	glcm_ClusterTendency	95
	shape_Sphericity	76
	Age	63
	T_status	49
	MTV2.5	40
	glcm_SumEntropy_MTV40	39
	shape_Elongation	37
	SUVmax	31
	Histology	29
1-year LRS	GTV	83
	Age	82
	glcm_ClusterTendency	65
	glcm_SumEntropy_MTV40	63
	shape_Sphericity	57
	Gender	48
	gldm_GrayLevelVariance	47
	N_status	27
	Stage	25
	glrlm_GrayLevelVariance	24
1-year DMS	GTV	99
	glcm_ClusterTendency	96
	shape_Sphericity	57
	MTV2.5	52
	Histology	34
	T_status	33
	Age	32
	glcm_SumEntropy_MTV40	32
	shape_Elongation	29
	SUVmax	23

Discussion

The rationale of this study was to identify a group of FDG PET radiomics features for NSCLC patients that are robust, independent, prognostic, and complementary to well-established clinical variables. We found PET radiomics features that met the study criteria of robustness and independence, and that also exhibited prognostic value. However, results demonstrated that PET radiomics features are not complementary to clinical variables for predicting clinical endpoints in NSCLC patients that were treated with CCRT. This indicates that clinical variables provide more prognostic information than robust independent PET radiomics features, and that the prognostic value in PET radiomics features is minimal. This study did take into account shortcomings of other studies on PET radiomics features [50] with the use of a feature selection method that reduces overfitting and external validation of results.

Feature selection based on the repeatability of PET radiomics features was feasible with the use of different phases from 4D PET imaging, in the absence of test-retest data. Larue et al. showed that in 4D CT, the majority of the features have a high agreement between radiomics feature stability based on 4D CT and test–retest data in lung cancer [67]. It was therefore hypothesized that 4D PET scans could also be used for repeatability testing. To determine robust PET radiomics features, a CR of 30% was chosen as limit for repeatability, based on PERCIST. However, a limitation of using 4D PET for repeatability testing is the absence of biological tumour variability, and PERCIST takes this variability into account. Hence, the use of a 30%-limit could be seen as too tolerant, and 15%, as commonly used in phantom studies, could be more appropriate. Even under these stricter circumstances, 12 first-order features, 24 out of 74 texture features, and all shape features would still meet that criterion as can be seen in S1 File. Besides that, the most prominent PET radiomics features in the fitted GLMs were SUVmax (CR = 13.2%), shape Sphericity (CR = 3.2%), GLCM ClusterTendency (CR = 21.9%), GLRLM GrayLevelNonUniformityNormalized (CR = 18.4%), and MTV2.5 (CR = 5.9%) as seen in Table 3. This shows that repeatable PET radiomics features with a CR>15% are also frequently present in the fitted models. Even though there is literature reporting on stability of PET radiomics features in a test-retest setting [45, 46], there is no objective limit for the level of repeatability for each PET radiomics feature. Determining such an objective limit is only relevant if the studied PET radiomics feature contains clinically useful information. Hence, in the absence of an objective limit for each PET radiomics feature, the 30%-limit of PERCIST was applied to all.

It was observed that the repeatability for features from MTV2.5 is better compared to MTV40 and this is due to two important factors:

From the 13 shape features, only MTV40 had a CR>30% when comparing the MTV40 between two mid-position scans. This variance, of course, has already a great impact on PET radiomics features calculated from MTV40 as it is known that differences in delineation have an impact on feature outcome [44].

Radiomics features are calculated on matrices which dimensions are dependent on the SUV range. With MTV2.5 matrix dimensions are more standardized than MTV40, which is dependent on the maximum SUV (CR = 13.2%).

In this case, the use of MTV2.5 for GTV delineation may be advised over MTV40 in PET radiomics analysis.

Another step of the feature selection procedure was to assess the independence of PET radiomics features, to identify possible prognostic features that could complement basic SUV metrics and volumetric features. In this context, changes in PET radiomics features would be independent from changes in basic SUV metrics and volume, increasing their utility in longitudinal studies. Therefore, the use of a fixed bin width for SUV binning should be avoided as this method resulted in PET radiomics features that were all strongly correlated to either maximum SUV or MTV. While the choice of |ρ|<0.5 for independence testing may seem arbitrary, a |ρ|<0.7 was also studied and did not improve results (see S1 File for more details). Independence testing had the most impact in the feature pre-selection procedure as it resulted in a substantial decrease of PET radiomics features. Unfortunately, results demonstrated that independence testing could not guarantee that remaining robust independent PET radiomics features exhibited complementary value next to clinical variables. Even so, we strongly advise assessing the relationship of radiomics features with current established prognostic factors in any study considering PET radiomics features for prognostication as this is the first important step in showing their potential added value in the clinic.

A final selection of features in the GLM was performed by elastic net regression, robust to collinearity amongst features [66]. More feature selection/classification methods exist [68], though comparing multiple methods was beyond the scope of this study. However, in literature, elastic net regression yielded one of the highest discriminative performances in chemoradiotherapy outcome prediction in 12 patient datasets containing in total 1053 lung cancer patients [65]. Interestingly, elastic net regression could also be used as a standalone feature selection method. A comparison of the feature selection method based on repeatability, independence, and elastic net regression (GLMall), and a method using only elastic net regression (GLMelnet) was performed, see S1 File. Pre-selection of PET radiomics features is worthwhile, because the number of PET radiomics features in GLMelnet was often high (>20 features) and many were highly correlated to volume or SUVmax. In contrast, the average number of features in GLMall was 9. Even so, it was observed that elastic net tends to keep all of the correlated and presumably prognostic features in the fitted model or shrinks all to zero, whereby increasing the number of (correlated) features resulted in a decrease of the predictive performance. This decrease of predictive performance seen in the validation set suggests that overfitting, although reduced, may still be present. This shows the value of dimensionality reduction in order to optimize predictive performance in rather small sample sizes.

The predictive performance of PET texture features in NSCLC has been studied widely, but clear evidence that PET texture features are complementary to clinical variables is lacking [69]. This study has extensively studied PET texture features and did not find any evidence for added value next to current clinical variables. S1 File provides a complete overview of all assessed model performances, including additional investigations with TLG. In literature, typically, only one or two PET texture features have been significantly associated with predicting various survival endpoints [39–41, 47, 70–72]. However, of all the prognostic PET texture features from those studies, such as GLCM Joint Entropy, Correlation, Contrast, Dissimilarity (or Difference Average), NGTDM Coarseness, Busyness, and Contrast, only GLCM Joint Entropy was both repeatable and independent from SUVmax or volume in our dataset. In this study, GLCM Joint Entropy was selected 34 times out of 100 by elastic net regression for predicting 2-year OS, and its value in overall survival was also previously shown [47]. Nonetheless, in our study the average predictive performance for GLMall in all clinical endpoints ranged from 0.50 to 0.66. For comparison, other studies predicting outcome with both PET radiomics and clinical variables in NSCLC found predictive performances of 0.63 for predicting OS [41], 0.72 for local recurrence [71], and 0.71 for distant metastases [72]. Even with those results, neglecting any limitations of those studies, there is still no strong evidence that PET texture features exhibit complementary information.

Results from the external validation demonstrated even lower AUC values in most cases than the internal validation set. Besides the limitation of the use of a small external dataset, differences were observed between institutes regarding patients, treatment, and image acquisition and reconstruction settings, that also can influence outcome [44, 73], and could have resulted in poor generalizability. To overcome the issue of poor generalizability, a prognostic model should be trained on a combination of well-balanced patient cohorts from multiple institutes, and PET acquisition and reconstruction protocols should be harmonized across centers in multi-centre studies. Alternatively, a post-reconstruction harmonization method proposed by Orlhac et al. may also aid in removing the multicenter effect for textural features and SUV [74].

Furthermore, limitations of this paper include the relatively small sample size for machine learning methods that could have affected the predictive performance [75], and the impact of tumour motion on PET radiomics features, especially in lower lobe tumours [76]. Although Grootjans et al. showed that there are specific PET radiomics features whose prognostic accuracy was not affected by respiratory motion and varying noise-levels [29].

To overcome the limitations of this study, and to be certain that there is no complimentary information in PET radiomics features, future studies need to set up large scale multi-centre cohorts to allow for multiple independent validation datasets. To further improve predictive performance, studies could investigate elastic net-Cox proportional hazard models [77], non-linear relationships by applying data transformation on PET radiomics features [21, 78, 79], or assess computer engineered features with neural networks or deep learning networks [80, 81]. Currently, deep learning is under investigation for use in lung nodule detection, tumour segmentation, and tumour classification with histopathology images [82]. Its use in medical image analysis is increasing as algorithms become more sophisticated and more data becomes available, which might lead to new insights in survival prediction. A step further would be to combine radiomics features from multimodal imaging such as PET, CT and MRI [83, 84], where the combination of anatomical and biological features may of added value for providing a personalized treatment strategy.

Conclusion

In this study, robust independent PET radiomics features, identified with 4D PET imaging, did not have complementary value in predicting overall survival and progression-free survival in NSCLC patients treated with concurrent chemoradiotherapy. Improving risk stratification and clinical decision making based on clinical variables and PET radiomics features has still been proven difficult in locally advanced lung cancer patients. New approaches should be investigated in large scale multi-centre studies to deal with current challenges in the field of radiomics before translation to the clinic becomes realistic.

(DOCX)

Click here for additional data file.

5 Dec 2019

PONE-D-19-30593

Robust, independent and relevant prognostic 18F-fluorodeoxyglucose positron emission tomography radiomics features in non-small cell lung cancer: are there any?

PLOS ONE

Dear Mr Konert,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

We would appreciate receiving your revised manuscript by Jan 19 2020 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Domenico Albano

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements:

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at http://www.plosone.org/attachments/PLOSOne_formatting_sample_main_body.pdf and http://www.plosone.org/attachments/PLOSOne_formatting_sample_title_authors_affiliations.pdf

We noticed you have some minor occurrence(s) of overlapping text with the following previous publication(s), which needs to be addressed:

2. In your revision ensure you cite all your sources (including your own works), and quote or rephrase any duplicated text outside the Methods section. Further consideration is dependent on these concerns being addressed.

3. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially identifying or sensitive patient information) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. Please see http://www.bmj.com/content/340/bmj.c181.long for guidelines on how to de-identify and prepare clinical data for publication. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

4.  Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.

Additional Editor Comments (if provided):

Dear author,

please follow reviewers suggestions to improve the paper.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The topic is of interest; The objective is not novel.

The methodology is appropriate and described in detail.

The authors report negative results: no added value of FDG-PET radiomics features in prediction of outcome in advanced NSCLC.

Overall, the study is of good quality. The results could contribute to the current debate on radiomics value Nonetheless, the findings may be dependent on the study conditions, including the patient cohort size, and, therefore, not provide definitive data.

Reviewer #2: This is a very timely paper addressing the robustness of PET radiomic features.

The method is sound and thorough. The paper is well-written and easy to follow.

Thus, only a few very minor comments:

1) As also mentioned by the authors the repeatability testing is not test-retest as is normally expected when discussing repeatability, but a comparison of 2 mid-positions scans generated from the same 4D PET/CT scan. Thus, only very little variance would be expected.

2) The repeatability appears better for MTV2.5 compared to MTV40 - could the authors please comment on this?

3) Typo on p12/235: "Only one? MTV40 was not repeatable"

4) The quality of the figures are not very high - at least when printed.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

19 Jan 2020

For the response to specific reviewer and editor comments, please see the attached file 'response to the reviewers'.

Submitted filename: Response to Reviewers.docx

Click here for additional data file.

24 Jan 2020

Robust, independent and relevant prognostic 18F-fluorodeoxyglucose positron emission tomography radiomics features in non-small cell lung cancer: are there any?

PONE-D-19-30593R1

Dear Dr. Konert,

We are pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it complies with all outstanding technical requirements.

Within one week, you will receive an e-mail containing information on the amendments required prior to publication. When all required modifications have been addressed, you will receive a formal acceptance letter and your manuscript will proceed to our production department and be scheduled for publication.

Shortly after the formal acceptance letter is sent, an invoice for payment will follow. To ensure an efficient production and billing process, please log into Editorial Manager at https://www.editorialmanager.com/pone/, click the "Update My Information" link at the top of the page, and update your user information. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, you must inform our press team as soon as possible and no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

With kind regards,

Domenico Albano

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

11 Feb 2020

PONE-D-19-30593R1

Robust, independent and relevant prognostic 18F-fluorodeoxyglucose positron emission tomography radiomics features in non-small cell lung cancer: are there any?

Dear Dr. Konert:

I am pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

For any other questions or concerns, please email plosone@plos.org.

Thank you for submitting your work to PLOS ONE.

With kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Domenico Albano

Academic Editor

PLOS ONE