Introduction to radiomics and radiogenomics in neuro-oncology: implications and challenges.

Neuro-oncology largely consists of malignancies of the brain and central nervous system including both primary as well as metastatic tumors. Currently, a significant clinical challenge in neuro-oncology is to tailor therapies for patients based on a priori knowledge of their survival outcome or treatment response to conventional or experimental therapies. Radiomics or the quantitative extraction of subvisual data from conventional radiographic imaging has recently emerged as a powerful data-driven approach to offer insights into clinically relevant questions related to diagnosis, prediction, prognosis, as well as assessing treatment response. Furthermore, radiogenomic approaches provide a mechanism to establish statistical correlations of radiomic features with point mutations and next-generation sequencing data to further leverage the potential of routine MRI scans to serve as "virtual biopsy" maps. In this review, we provide an introduction to radiomic and radiogenomic approaches in neuro-oncology, including a brief description of the workflow involving preprocessing, tumor segmentation, and extraction of "hand-crafted" features from the segmented region of interest, as well as identifying radiogenomic associations that could ultimately lead to the development of reliable prognostic and predictive models in neuro-oncology applications. Lastly, we discuss the promise of radiomics and radiogenomic approaches in personalizing treatment decisions in neuro-oncology, as well as the challenges with clinical adoption, which will rely heavily on their demonstrated resilience to nonstandardization in imaging protocols across sites and scanners, as well as in their ability to demonstrate reproducibility across large multi-institutional cohorts.

Radiomics has recently emerged as a powerful data-driven approach that can offer insights into clinically relevant questions related to diagnosis, prediction, prognosis, as well as assessing treatment response.

Using radiomics and radiogenomic approaches to personalize treatment decisions in neuro-oncology will rely heavily on resilience to nonstandardization in imaging protocols across sites and scanners, as well as in their ability to demonstrate reproducibility across large multi-institutional cohorts.

The field of neuro-oncology encompasses primary and metastatic malignant tumors of the central nervous system (CNS) including the brain and spinal cord. While the incidence of brain tumors is rare (lifetime likelihood of less than 1%), their mortality and morbidity rate is unusually high. In 2020, around 23 890 cases of malignant brain tumors and 18 020 brain cancer-related deaths are estimated in adults across the globe.[1,2] Among the malignant tumors, brainstem gliomas, glioblastomas (GBMs), and anaplastic astrocytomas have the worst outcome.[3,4] While the survival rate for patients with a malignant brain or CNS tumor is poor with only 36% living beyond 5 years following diagnosis, it is particularly abysmal in GBM patients (age group of 55–64 years) who have a 5-year survival of 6%. Unfortunately, at this time, there are no widely recommended tests to screen for brain and spinal cord tumors.

Diagnosis, and treatment response assessment, in neuro-oncology is currently investigated using multi-parametric magnetic resonance imaging (MRI) such as T1-weighted imaging both before (T1w) and after administration of gadolinium-based contrast agent (Gd-T1w), T2-weighted imaging (T2w), and T2w-Fluid attenuation recovery (FLAIR) sequences. For instance, Response Assessment in Neuro-Oncology group (RANO criteria) utilizes the increase/decrease in the product of perpendicular diameter on routine MRI to identify patient’s response to treatment. On routine multi-parametric scans, Gd-T1w MRI is useful for elucidating a brain tumor, with large portions of the tumor region (or the entire tumor) typically enhancing in comparison to the brain parenchyma. Meanwhile, T2w and T2w-FLAIR MRI scans are often used to identify heterogeneously enhancing tumors like gliomas especially GBMs which often have peri-tumoral edema or inflammation, in order to differentiate these regions from the enhancing tumor.

In addition to qualitative/semiquantitative assessment of these imaging modalities by trained neuro-radiologists, radiologic imaging also harbors massive amounts of interpretable quantitative information which may not be appreciable via visual inspection or 2-dimensional assessment of a single measure (ie, tumor diameter).[5] Recent advances in the high-throughput computational techniques and rapid algorithm developments have facilitated the extraction of meaningful information from radiologic imaging via “Radiomics.” Radiomics is defined as the extraction of “hand-crafted” features from routine radiological scans (X-rays, CT, MRI, and PET) that quantitatively capture the textural and morphological characteristics of a given tumor. For instance, radiomics attempts to comprehensively characterize the pixel-wise tumor characteristics including (1) semantic or qualitative features that include radiologist-derived assessments of the tumor including spiculations, size of the tumor along several axes, (2) shape-based features that quantitatively measure regular or irregular tumor boundary changes based on their 3D topology,[6] (3) intratumoral heterogeneity measures including gray-level features, which investigate pixel-level textural differences to characterize how heterogeneous a tumor is,[7,8] as well as (4) deformation features that capture the impact of tumor-related mass effect in the tumor micro-environment.[9] These various radiomics-based approaches have been widely employed across multiple cancers including brain,[10-14] lung,[15-18] colorectal,[19-21] breast,[22-24] and prostate[25,26] among others for diagnosis, prognosis, treatment response prediction, as well as measuring early treatment effects. Specifically, in the context of neuro-oncology, these approaches have shown tremendous promise in the development of noninvasive radiomic prognostic and predictive markers, as well as for distinguishing treatment effects from tumor recurrence, using routine MRI scans.[10,27-29]

In addition to advances in multi-parametric imaging, next-generation sequencing technology, which allows for the high-throughput sequencing of multiple genes, has been a significant propeller of tailoring personalized treatments in neuro-oncology.[30,31] Recent investigative studies have identified several driver mutations and chromosomal anomalies as specific targets for personalizing treatment and improving prognosis[32,33] in neuro-oncology. A few of these driver mutations have been reported to have prognostic (eg, isocitrate dehydrogenase [IDH], O6-methylguanine-DNA methyltransferase [MGMT] promoter methylation, and epidermal growth factor receptor [EGFR]) and predictive (eg, MGMT) implications in GBM tumors.[34,35] However, molecular profiling involves tissue extraction often from stereotactic/needle biopsies that are inherently prone to sampling bias based on the site of biopsy and thus may not capture the spatial heterogeneity extant within the tumor (specifically in highly heterogeneous tumors such as GBM).[36]

Interestingly, the field of “radiogenomics” has provided a mechanism for establishing statistical associations of tumoral radiomic features with the underlying genetic profile of the tumor, including point mutations, signaling, and pathways of biological significance. By providing an imaging phenotype for the tumor corresponding to a particular genotype, radiogenomics may allow for circumventing the challenges with biopsy sampling. For example, radiogenomic approaches have been used to create “virtual-biopsy” maps to predict specific prognostic point mutations as well as chromosomal alterations[37] based on the 2016 update on the WHO classification of diffuse gliomas in neuro-oncology.[13,38,39] Identifying such radiogenomics associations may improve our understanding of how the changes in biological processes at the molecular level affect changes at a radiologic scale[40] and may ultimately aid in personalizing treatment decisions.

Overview of Radiomic and Radiogenomics Pipeline

One of the primary advantages of developing and implementing a radiomic/radiogenomics pipeline is that, ideally, by leveraging routine MRI scans, the analysis does not significantly disrupt the existing clinical workflow. A typical pipeline of a radiomic/radiogenomics-based approach consists of the conversion of radiographic images into mineable data and involves the following steps: (1) image acquisition and registration, (2) segmentation of region of interest, (3) preprocessing, (4) feature extraction, (5) feature selection and building machine learning models for predictive and prognostic applications, and lastly (6) radiogenomic associations to either predict a genotype or identify the biological processes that drive the tumor biology. Figure 1 shows the complete radiomic/radiogenomics pipeline.

Figure 1.

Overall workflow of radiomic and radiogenomic pipeline.

Image Acquisition, Tumor Segmentation, and Preprocessing

Preprocessing is a key step prior to radiomic feature extraction and involves multi-protocol registration to account for patient movement during image acquisition, as well as accounting for image variations in MRI scans across different manufacturers and multiple participating sites. Next, the region of interest is identified and segmented, either manually or automatically. On Gd-T1w images, necrosis is relatively represented as hypo-intense regions that are commonly located in the central region of the tumor and occasionally have a ring enhancement. Similarly, hyper-intense T2w-FLAIR signals correlate with greater interstitial leakage and low cellular density, reflecting edema. Therefore, T2w and T2w-FLAIR scans are typically used to identify edema and necrosis and enhancing tumor is delineated based on Gd-T1w MRI. The “tumor habitat” consisting of the 3 segmented tumor subcompartments, necrotic core, enhancing, and peri-tumoral edema, can then be interrogated by extracting radiomic features from each of the tumor subcompartments across different MRI protocols. Figure 2 illustrates the tumor habitat in GBM as delineated by an expert radiologist using routine MRI protocols. Currently, manual segmentations are considered the gold standard for radiomic analysis as they ensure high accuracy. However, there exist several automated segmentation approaches using deep learning (DL) architectures including U-Net, Conv-Net, Transfer Learning, and Deep Hourglass approaches as well as semiautomatic seeding-based algorithms that have gained popularity.[41-43] These automated tools have shown promise in annotating the tumor subcompartments including enhancement, nonenhancement, necrosis, as well as peri-tumoral edema from the immediate periphery of the tumor.

Figure 2.

Using multi-parametric MRI scans to identify the entire tumor habitat of glioblastoma as delineated by an expert radiologist. Necrotic core (as marked in green) and enhancing tumor (as marked in blue) can be identified on post-contrast T1w MRI scans. Similarly, the peri-tumoral edema (as marked in yellow) can be identified on the T2w/T2w-FLAIR MRI scans.

Further preprocessing techniques such as skull stripping and intensity standardization are executed to account for varying magnetic strengths, and slice thicknesses in the curated dataset.[44-47] Intensity standardization algorithms are implemented across multi-parametric MRI protocols to correct for intensity nonstandardness, which refers to the issue of MRI “intensity drift” across different imaging acquisitions. The intensity nonstandardness results in MRI intensities lacking tissue-specific numeric meaning within the same MRI protocol, for the same body region, or for images of the same patient obtained on the same scanner. Therefore, it is important that the radiomic pipelines in neuro-oncology implement the necessary preprocessing techniques to ensure consistency in the dataset and reproducibility of results.[48]

Radiomic Feature Extraction

Following preprocessing and lesion segmentation, quantitative features are extracted for radiomic analysis, via algorithms that capture tumor heterogeneity across local pixel neighborhoods within a given radiologic image. Common radiomic features are currently divided into the following classes: semantic, shape, texture/gradient-based, deformation, and wavelet. Figure 3 illustrates an example of a few radiomic features extracted from the multi-parametric MRI scans from across the GBM tumor habitat. The specifics of different feature families are discussed in the subsequent subsections.

Figure 3.

(Top Row) Multi-parametric MRI is used to annotate the tumor subcompartments in GBM patients. The GBM tumor habitat consists of (1) necrotic core, (2) enhancing tumor, and (3) edema. Haralick features such as entropy and IDM are extracted from the GBM tumor habitat. (Bottom Row) Deformation features are calculated from the brain around tumor region. CoLlAGe gradients detect the homogeneous and heterogeneous regions based on the local intensity patterns on MRI scans.

“Hand-crafted” radiomic descriptors

Semantic features

In a large National Cancer Institute’s The Cancer Genomic Atlas (TCGA)-based effort, brain tumors (GBMs) were examined by experienced neuro-radiologists to define “semantic” features that capture the visual phenotypic characterization of the tumors. These unique semantic features were based on the 4 commonly analyzed tumor subcompartments (nonenhanced tumor, enhancing tumor, necrosis, and edema) and included features such as location of the lesion, morphology, major axis length, minor axis length margin, and lesion vicinity. This comprehensively characterized feature set by expert radiologists is now known as Visually AcceSAble Rembrandt Images (VASARI) features.[49] Multiple studies have demonstrated that prognostic imaging attributes, such as percent of necrosis and edema, can be correlated with VASARI features (ρ = 0.67 [P < .001] and ρ = 0.41 [P < .001]) in GBM tumors.[50,51] Another study found a significant correlation between overall survival (OS) in GBMs and VASARI features. These studies have established that VASARI features (such as degree of contrast enhancement as well as the length of the major axis of the lesion) may provide additional prognostic value on top of standard clinical variables (such as Karnofsky Performance Score [KPS]).[52] Several radiogenomic-based studies have similarly demonstrated that genetic expression or RNA sequencing of GBMs may be correlated with the VASARI features.[53-55] For instance, Diehn et al.[55] have presented an extensive study that seemed to suggest that GBM tumor contrast enhancement may be associated with activation of specific hypoxia gene expression.

Shape-based features

Irregular and aggressive tumor infiltration may prompt surface and shape changes in the tumor and peri-tumoral regions. A larger surface-area-to-volume ratio, for example, has been found to be indicative of a more spiculated tumor, with more malignant potential in comparison to a round mass with a smaller ratio.[56] Driven by this intuition, shape features may be informative of tumor malignancy. These features are broadly classified into 2 categories—local and global features. Local surface features capture characteristics such as curvature that identifies flat areas of surface from highly curved ones, sharpness that measures how sharp the curvature is, with highly curved masses exhibiting sharper curvatures, as well as shape index that characterizes the shape topology of the tumor.[15] Whereas some of the global shape features include the major and minor axis, and elongation (ratio between major and minor axes of the region of interest). Several studies have shown that shape-based features of the tumor subcompartments in GBMs may be used to predict OS.[8,57-60] Gevaert et al.[51] have shown that high irregularity of enhancing lesion shape (ie, degree of spiculation) was associated with a lower OS. In a radiogenomic setting, Itakura et al.[61] have reported that shape-driven features can classify GBMs into 3 distinct clusters, where each cluster is mapped to a unique set of molecular signaling pathways and differential probabilities of survival. For instance, they reported that cluster 2 (characterized by spherical tumors with regular edges and small volumes) was associated with downregulation of VEGFR signaling pathway. Apart from this, shape features have also been shown to be better in evaluating treatment response. Ismail et al.[6] have demonstrated that 2 local features (total curvature of the enhancing lesion and curvedness of the T2w/T2w-FLAIR hyper-intense peri-tumoral edema) may be able to differentiate between pseudo-progression and tumor progression in GBMs.

Texture/gradient-based features—

The most commonly used texture features include (1) gray-level co-occurrence matrix (GLCM) or Haralick features that capture the variations in gray-level image characteristics via second-order intensity statistics (eg, entropy, inverse differential moment [IDM], angular second moment, contrast, and differential entropy).[62] Haralick features potentially capture the structural heterogeneity within the region of interest. For instance, IDM is a reflection of the presence or absence of uniformity and hence is a measure of local regions of homogeneity. (2) Gray-level run length matrix (GLRLM), which in comparison with GLCM features, investigates the pixel runs instead of pairs of pixels. A pixel run includes the number of pixels of a specific gray value that are in a right direction, in the right sequence. While the rows of the matrix still represent gray levels, the columns represent run lengths. (3) Laws features that define various texture parameters including spot, edge, ripple, and level surfaces present within the tumor[63] and (4) Co-occurrence of Local Anisotropic Gradient Orientations (CoLlAGe), which captures local anisotropic differences that exist in micro-structures by measuring entropy (a mathematical construct to measure disorder) of co-occurrences of pixel/voxel-level gradient orientations computed within a local neighborhood.[64]

Haralick features have been used extensively in the context of neuro-oncology to predict survival.[7,12] Kickingereder et al.[65] have shown that Haralick features extracted from the T2w-FLAIR MRI sequences can predict survival and stratify patients with newly diagnosed GBM. In 2 independent studies, it has been demonstrated that GLRLM features, extracted from multi-parametric MRI scans, are also prognostic of OS in GBMs.[66,67] GLRLM features extracted from 18F-FDG-PET have also shown to distinguish primary CNS lymphoma from GBMs.[68] In a cohort of 42 patients, CoLlAGe has been shown to differentiate radiation necrosis, a benign yet confounding effect of radiation treatment, from recurrent tumors, with an accuracy of 83.79% in primary cases, and 88.52% in metastatic brain tumors.[64,69]

Deformation-based features

Deformation features seek to measure the extent of tissue deformation in the brain parenchyma (ie, brain around tumor [BAT]) due to the mass effect in brain tumors. MRI scans are nonrigidly registered to equivalent healthy, age-matched, and/or gender-matched imaging atlases. Then, the resulting deformation field (represented as a displacement vector at every voxel location) is obtained through a combination of forward as well as inverse mapping between the patient’s 3D volume and the reference atlas. The per-voxel deformation measurements from the BAT region are then used as radiomic features for analysis. Prasanna et al.[70] hypothesized that larger variations in deformation within functional areas of the contralateral hemisphere are likely related to decreased survival in GBMs. They demonstrated that decreased OS was found to be associated with increased deformation in areas of language comprehension, visual perception, cognitive, and motor-control functions, particularly in the memory areas in the left hemisphere. In another study, they also showed that combining textural-based radiomic features with deformation values may result in improved prediction of survival in GBMs, for identifying short-term survivors (OS <240 days) from long-term survivors (OS >540 days).[9] In a radiogenomic setting, Iyer et al.[71] have shown that deformation-based radiomic features can also be used to differentiate the molecular subtypes of medulloblastomas (Sonic Hedgehog [SHH], Wingless [WNT], Group 3, Group 4) in pediatric brain tumors.

Wavelet-based features—

Wavelet features utilize different wavelengths, amplitudes, and frequencies to recognize textural attributes across a wide range of scales within the image. For instance, Gabor wavelet features are the steerable class of gradients that attempts to match localized frequency characteristics.[72] A Gabor filter can be defined as the modulation of a complex sinusoid by a Gaussian function. Each descriptor quantifies response to a given Gabor filter at a specific frequency (f = 0, 4, or 16) and orientation (µ = 45°, 90°, 135°, 180°) and attempts to capture the prominent direction in which intensity changes occur. Tixier et al.[73] have reported that GBM patients with large negative skewness of Gabor wavelets had a significantly longer median OS of 22.7 months (P = .004). In another study, a wavelet scattering-based noise robust radiomic method was implemented to predict the gliomas grade in brain tumors.[74]

Feature Selection and Building Prognostic and Predictive Models

Building radiomic-based machine learning classifiers involves reducing the high-dimensional radiomic feature set by selecting the most discriminative features using a feature selection scheme, in order to reduce the curse of dimensionality.[75] Feature selection can either be using univariate or multivariate statistical models. Univariate methods (also known as filter methods) only depend on feature association, while ignoring redundancy, whereas multivariate methods (wrapper methods) inspect inter-actions within different features and account for both association and redundancy. Fisher score, Chi-squared test, and Wilcoxon are the most common filter methods that are used for feature selection.[16] Wrapper models are computationally expensive as their objective is to find a subset of features that result in the best performing model. Notable examples for wrapper methods include forward feature selection, backward feature elimination, exhaustive feature selection (greedy algorithm), or bidirectional search.[16,76]

After identifying a subset of features following feature selection, machine learning models are developed by categorizing and classifying various datasets according to defined labels (ie, GBM vs low-grade glioma [LGG], poor survival vs improved survival, radiation effects vs tumor recurrence). Generally, classifiers can be divided into supervised and unsupervised approaches. While supervised methods utilize a predefined set of known labels to identify features that best represent the outcomes of interest on the radiologic images, an unsupervised approach (such as clustering) can be employed when the target labels are unknown. A range of classifiers including random forest, support vector machines (SVMs), and generalized linear models have been used for diagnosis and treatment response evaluation applications.[6,12,76] For instance, previous studies have used SVM classifier to predict the histopathological grade (LGGs vs GBMs) of a given primary brain tumor using MRI scans.[76-78] Research groups have employed least absolute shrinkage and selection operator (LASSO) logistic regression and SVM models[79,80] to differentiate pseudo-progression from early tumor progression in GBM patients. Prasanna et al.[7] have shown that radiomic features extracted from the peri-tumoral edema on multi-parametric MRI were able to distinguish short-term survivors (OS <7 months) and long-term survivors (OS >18 months) in 65 GBMs studies. Recently, DL-based architectures such as convolutional neural networks and auto-encoders have also been used to extract features from the radiologic imaging data.[27,81] While DL is capable of apprehending more abstract and higher-dimensional relationships between imaging features and the clinical end point, these approaches are still largely considered black-box.[82]

Survival Analysis

In most of the survival-based (ie, prognostic) radiomic/radiogenomic studies, Cox proportional hazards model are used, where the event of disease diagnosis is chosen as the time of origin. The commonly used end points for survival analysis are either OS (time from disease diagnosis to death due to the cancer in question) or time duration from a given treatment to response or outcome (progression-free survival [PFS]).[83] In a Cox proportional hazard regression model, hazard rate (HR) quantifies the effect of individual feature on survival and measures the risk of failure (ie, the risk or probability of suffering the event of interest), given that the patient has survived up to a specific time. Risk parameters yielding negative regression coefficients (ie, low feature values correlated with long-term survival) produce a HR between 0 and 1; features yielding positive regression coefficients (ie, low feature values correlated with short-term survival) produce a HR between 1 and infinity.[40,84,85]

Kaplan–Meier (KM) curves, a nonparametric survival analysis method, have also been well accepted for evaluation of the “time-to-event” data.[86] KM curves evaluate the probability of survival over time (dependent variable), under different conditions (independent variable), where the horizontal axis represents the time and the vertical axis shows the probability of survival. After the KM curves have been plotted, the log-rank test is used to compare the 2 curves (high risk vs low risk). The log-rank test calculates the chi-square (χ 2) for each event time across all groups and sums up the results. χ 2 from the log-rank test concludes whether 2 curves from the groups are statistically different.[87]

Radiogenomic Studies

Multiple radiomic features have been shown, via radiogenomic analysis, to capture genomic alterations within tumor DNA, on routine MRI scans. These radiomic features are shown to identify the presence of specific mutations that have implications in the management and outcome of neuro-oncology patients.[54,88,89] For instance, several groups have demonstrated that the IDH mutations in diffuse gliomas may be predicted via radiomic signatures on pretreatment MRI (Figure 4).[32,38,90,91] In a recent study, Shboul et al.[92] have demonstrated that radiomic features can discriminate IDH mutation from IDH wild type in LGG with an AUC of 0.84. They presented that higher values of the size ratio between the enhancing tumor and the necrosis, and higher values of the vertical orientation of edema major axis were significantly associated (ANOVA test, P value < .005) with IDH wild-type status. Similarly, radiogenomic models have also been developed to predict other imaging biomarkers for MGMT,[13,33,73] EGFR,[93] TERT,[94,95] PTEN,[96] and ATRX[97] mutations in neuro-oncology.

Figure 4.

Isocitrate dehydrogenase (IDH) is an independent prognostic factor in gliomas, with mutated IDH1 and IDH2 having improved prognosis compared to gliomas with wild-type IDH. Gradient and intensity statistics texture features within edema in our preliminary work were found to discriminate IDH1 mutation versus wild-type gliomas on n = 78 studies.90

Furthermore, radiogenomic analysis could be leveraged to elucidate the underlying biological basis of the prognostic radiomic features, by identifying signaling pathways that drive tumor biology between the radiomics-driven survival groups.[40,98,99] These radiogenomic approaches first identify the prognostic radiomic features that can identify low-risk from high-risk groups based on patient outcome (OS or PFS). Next, bioinformatics tools such as Gene Ontology (GO)[100,101] and single-sample gene set enrichment analysis (ssGSEA)[102] are employed to detect associated downstream signaling pathways of biological significance across the low-risk and the high-risk categories. For a predefined set of genes, ssGSEA captures the significantly enriched or depleted biological processes and calculates an enrichment score for every patient in the cohort. GO is a knowledge base of the functions of genes and thus forms a foundation for computational analysis of large-scale molecular biology and genetics experiments. GO also highlights the most overrepresented genes and finds the systematic linkages between those genes and biological processes.

Using a radiogenomic approach, Liu et al.[11] demonstrated that PFS in LGG may be noninvasively predicted using radiomic features from T2w MRI scans, and then by using GO analysis, revealed that their identified prognostic radiomic features were associated with biology processes of programmed cell death and cell proliferation. Another radiogenomic study developed and independently evaluated a LASSO-based Radiomic Risk Score (RRS), using radiomic features from the tumor habitat, to stratify 203 GBM patients into low-risk and high-risk groups based on PFS. The RRS consisted of a total of 25 radiomic features including textural features belonging to Laws energy and Gabor wavelet families, and shape features from the peri-tumoral edema region, on Gd-T1w MRI. The study then identified significant radiogenomic correlations (P < .05) between the prognostic radiomic features with molecular signaling pathways, such as cell differentiation, cell adhesion, and angiogenesis, using GO and ssGSEA.[40] Thus, by establishing multi-scale associations of radiomic phenotypes with corresponding transcriptomic data, radiogenomics approaches may allow for an improved understanding of the underlying disease biology as well as potentially aid in build patient-centric treatment plans.

DL-Based Approaches

Unlike traditional radiomic-based approaches that rely on accurate segmentation and selection of hand-crafted engineered features, DL methods are domain-agnostic. Most DL approaches do not require labor-intensive segmentations and have the ability to capture complex hidden visual representations of radiologic data.[82,103-105] Convolutional neural networks (CNNs) are the most common DL models used in medical image analysis.[106] A CNN has an input layer, an output layer, and the parallel computations are performed within multiple hidden layers, which include convolutional and pooling layers. Convolutional layers are the building blocks of CNNs from which features are extracted from the input layer (to which small patches of image data are fed). Pooling layers transfer the maximum or average convolved values and consequently reduce the size of the features extracted. Repeated mathematical operations of convolution and pooling result in altered representations of the imaging data and capture several features including (but not limited to) edge detection, color variations, sharpening, blurring, and focusing. During the training phase, guided by a loss function (which estimates the difference between the labels and predictions), the CNN determines the weights of these convolution filters in recognizing subtle visual signatures hidden in images.

In the context of brain tumors, Lao et al.[81] developed a DL-based radiomics model using Gd-T1, T2w, T2w-FLAIR MRI protocols and clinical data (age and KPS) from 112 GBM patients and obtained a C-index of 0.71 in predicting OS. In a recent study, Bae et al.[107] demonstrated that a deep neural nets can distinguish GBMs from metastasis with an accuracy of 0.95 (95% CI, 0.92–0.99). In another study by Han et al.,[108] a bi-directional recurrent CNN on 260 TCGA-GBM patients was developed to obtain an accuracy of 62% in predicting MGMT gene status on an independent test data using multi-parametric MRI.

Limitations

While highly promising, a key challenge in enabling clinical utility of radiomic/radiogenomic approaches is to demonstrate their generalizability to variations in image acquisition protocols across scanners and sites. Sources of variations in MRI acquisition often include differences in image contrast, voxel resolutions, slice thicknesses, image reconstruction methods, magnetic field strengths, echo times, and repetition times. This issue of reproducibility of radiomic features has become even more pertinent in retrospective studies that involve publicly available repositories such as TCGA-GBM and Ivy GAP, where image scans are pooled-in from different institutions.

Several efforts are currently underway to fill these technical gaps, including attempting to standardize the image acquisition, preprocessing, segmentation guidelines, and radiomic features extraction pipelines across multiple sites.[109] Most recently, the image biomarker standardization initiative has provided best-practice guidelines for standardizing feature extraction pipelines from MRI scans from different sites and scanners.[110] Additionally, open-source software platforms such as Cancer Imaging Phenomics Toolkit, which is specifically developed for neuro-oncology applications[111] as well as a more generalized Pyradiomics[112] platform, also aim to provide the medical community with standardized set of pipeline for radiomic feature analysis. Additionally, a few recent studies[113-115] have explored the issue of repeatability, which refers to the variability in radiomic features across scans obtained at 2 different times on the same scanner, as well as reproducibility, which is defined as the variations in radiomic features on account of differences in image acquisition across sites and scanners. These repeatability and reproducibility radiomic studies have been conducted by leveraging test–retest datasets or phantom studies with varying imaging acquisition protocols. However, rigorous analysis is warranted to improve the generalizability of radiomic features, in terms of repeatability, reproducibility as well as efficacy, across large multi-site cohorts (preferably using retrospective clinical trial datasets). Another important aspect of the radiomic pipeline is the segmentation of the tumor habitat. Manual tumor segmentation is not only labor intensive, but is also affected by inter-observer variability.[5,16] While some radiomic studies use automatic and semiautomatic methods for segmentation, the existing segmentation algorithms are not consistent among different research groups, and may further have an impact on the radiomic analysis as well as downstream prognostic and predictive analysis.

Similarly, while the advent of DL networks has opened new avenues of research in GBM analysis, it comes with its own set of limitations. Unlike the radiomic features that often provide at least some degree of interpretability, DL features are considered more of a “black-box”.[116] These deep features are limited in their explanatory capacity with neither a set of diagnostic rules nor an insight into the results. Additionally, DL models are limited by the relative sparsity of training samples, which is even more relevant in rare cancers such as GBMs, where obtaining very large data cohorts may not be feasible. Table 1 briefly contrasts the advantages and disadvantages of expert-based, radiomics and DL-based approaches in the context of brain tumor characterization.

Table 1.

Comparison of Radiomics and Deep Learning-Based Approaches

Expert-Based Evaluation		Radiomics		Deep Learning
Advantages	Limitations	Advantages	Limitations	Advantages	Limitations
Observation-driven	Qualitative/semiquantitative	Hand-crafted engineered features	Impacted by variance in image acquisition parameters introduced across sites and scanners	Domain agnostic data-driven	Known as “black-box” due to limited biological interpretability offered the deep features
Experience-driven	Labor intensive		Often dependent on segmentation of the tumor habitat
Low computational costs	Intra- and inter-observer variability	Hand-crafted engineered features	Often used for small retrospective data and may not be generalizable	Does not require segmentation of tumor habitat	Limited by relative sparsity of training samples, not always suited for applications with limited availability of well-curated samples
Abundant historical literature	Poor reproducibility

Conclusion and Future Scope

Radiomic, radiogenomic, and DL studies have made notable progress in the last few years and have demonstrated potential in the field of neuro-oncology, including aiding in diagnosis, outcome prediction, as well as evaluating response to both conventional and experimental treatments.[80,89,97] However, for clinical deployment of these approaches, a few important considerations need to be accounted for. First, it may be important to establish causal inference of the radiogenomic associations between radiomic features and the underlying tumor biology, either through controlled preclinical models or spatially co-localized imaging and -omics datasets.[117] Second, rigorous repeatability, reproducibility, and efficacy analysis of these approaches will need to be conducted across large multi-site retrospective cohorts. These large multi-institutional retrospective studies could then pave the way for prospective randomized trials to evaluate the efficacy of radiomic and radiogenomic markers in predicting response to treatment and guiding treatment decisions. Lastly, for translation of radiomic/radiogenomic/DL models as decision support in a clinical setting, careful planning will be needed in order to integrate the human and machine interpretations together and improve diagnostic and prognostic reads. Hence, unified efforts from all stakeholders including neuro-radiologists, neuro-oncologists, neuro-surgeons, along with data scientists, and machine learning researchers will be required toward development of these models from conception to deployment. Such cross-disciplinary collaboration will ensure that the tools being developed are tailored and aligned to benefit the patient in a variety of clinical settings including cancer screening, diagnosis, prediction of prognosis, and evaluating treatment response in neuro-oncology.

Funding

Research reported in this publication was supported by the National Cancer Institute of the National Institutes of Health under award number NCI 1U01CA248226-01, the DOD Peer Reviewed Cancer Research Program W81XWH-18-1-0404, Dana Foundation David Mahoney Neuroimaging Program, the V Foundation Translational Research Award, the Ohio Third Frontier Technology Validation Fund, and the Wallace H. Coulter Foundation Program in the Department of Biomedical Engineering.

Conflict of interest statement. N.B. is an employee of Tempus Labs, Inc. K.B. is a resident at Maimonides Medical Center, Brooklyn, New York. No potential conflicts of interest were disclosed by the other authors.