Radiomics in immuno-oncology.

With the ongoing advances in imaging techniques, increasing volumes of anatomical and functional data are being generated as part of the routine clinical workflow. This surge of available imaging data coincides with increasing research in quantitative imaging, particularly in the domain of imaging features. An important and novel approach is radiomics, where high-dimensional image properties are extracted from routine medical images. The fundamental principle of radiomics is the hypothesis that biomedical images contain predictive information, not discernible to the human eye, that can be mined through quantitative image analysis. In this review, a general outline of radiomics and artificial intelligence (AI) will be provided, along with prominent use cases in immunotherapy (e.g. response and adverse event prediction) and targeted therapy (i.e. radiogenomics). While the increased use and development of radiomics and AI in immuno-oncology is highly promising, the technology is still in its early stages, and different challenges still need to be overcome. Nevertheless, novel AI algorithms are being constructed with an ever-increasing scope of applications.

Introduction

Radiological imaging plays a vital role in the detection of cancer,, disease staging3, 4, 5 and monitoring treatment response., With emergence of the RECIST criteria in 2000, a standardized set of rules was constructed that enabled clinicians to better categorize tumour response during treatment. The field of oncology, however, is changing rapidly, primarily driven by increasing understanding of the underlying tumour biology. The emergence of novel immunotherapy using immune checkpoint inhibitors and genetically driven targeted therapy has improved the prognosis of several tumour types dramatically., While the clinical benefits provided to patients by these new biological treatments is undeniable, medical imaging has faced a new set of challenges in attempts to characterize the morphological patterns of response, progression and adverse events to targeted therapy and immunotherapy.11, 12, 13 Classical RECIST criteria, for example, proved suboptimal in the context of immunotherapy as patients treated with checkpoint inhibitors responded very differently from patients who received conventional chemo(radio)therapy. This shortcoming led to the creation of the iRECIST guidelines as a possible solution. The iRECIST criteria were undoubtedly steps in the right direction, especially for patients who would be classified as progressive disease under previous RECIST criteria. However, considering that only 20%-40% of patients respond to immunotherapy, there is a pressing need for reliable predictive biomarkers.

In comparison with other diagnostic modalities (e.g. pathological biopsies), imaging offers longitudinal insight into the patient's condition. This feature has brought radiology to the centre of therapeutic response and adverse event monitoring. Alongside the abovementioned iRECIST criteria, a number of studies have examined the use of positron emission technology (PET) imaging to assess response (e.g. PECRIT/PERCIMT criteria) and detect immune-related side effects.17, 18, 19, 20 Outside the scope of this review, it is believed that molecular imaging will prove undoubtedly helpful as the field of oncology moves towards precision medicine.

Significant advances in oncologic imaging have been witnessed in the quantitative aspects of radiology, particularly the domain of imaging features. Radiomic research is currently enjoying a rapid boom to identify non-invasive imaging markers/phenotypes capable of being linked with clinical/biological outcomes. The aim of this review is to provide a broad overview of imaging features, primarily radiomics, and highlight applications relevant to immunotherapy (response prediction) and targeted therapy (radiogenomics).

Imaging features

Imaging features, obtained from either anatomical or functional images, can be broadly divided into qualitative (or semantic) features and quantitative features. Semantic features are acquired by an experienced reader (i.e. radiologist) who scores different tumour characteristics (lesion shape, number of lesions, lesion intensity, etc.) during the evaluation of medical images. Quantitative features are derived by applying advanced mathematical algorithms to the images (as in the case of radiomics).,, Semantic and quantitative features can be used in statistical or artificial intelligence (AI) models to predict specific clinical endpoints. Individual radiomic features can be combined to form imaging ‘signatures’ or ‘phenotypes’. Table 1 provides an overview of the three major types of imaging feature approaches: semantics, handcrafted radiomics and deep learning radiomics.

Table 1

A key summary of the advantages and disadvantages of the different features

	Advantages	Disadvantages
Semantics	- More easily understandable to humans - Many semantic features are already included in the radiological workup of specific tumour types	- Requires an experienced reader to generate and interpret - Vulnerable to inter- and intra-observer variability - Limited (dimensionality)
Handcrafted radiomics	- Many features represent intuitive morphological features - Features encode morphological information beyond the limits of the human eye - Clear process pipeline - When the feature extraction is performed expertly, artificial intelligence trained on handcrafted radiomic features can perform just as well as deep learning, especially in smaller datasets - Requires less data than deep learning	- Algorithms contain human bias - Delineation is required - Influenced by different parameters (scanning equipment, pre-processing, scanning protocol)
Deep learning radiomics	- Order of magnitude more features - No pre-engineered algorithms - Often no expert delineation required - Can create automatic segmentation - Fully automated - Greater accuracy in specific tasks compared with traditional computer vision techniques	- Requires a significant number of samples for training - Publically available high-quality well-annotated data in medicine is scarce - Black box

Semantic features

Semantic features reflect immediately intuitive and often clinically reported image properties such as lesion shape, boundary type, density characteristics, intensity and others (see Figure 1).,,23, 24, 25, 26 These features, or image properties, provide additional information to the radiologist during the detection and diagnosis of disease as well as during the monitoring of response to treatment. Semantic traits were proven to be useful in non-invasive discrimination between malignant and benign computed tomography (CT) lung nodules. In non-small cell lung cancer (NSCLC), semantic features could differentiate between patients with epidermal growth factor receptor (EGFR), anaplastic lymphoma kinase (ALK) and Kirsten rat sarcoma viral oncogene homologue (KRAS) mutations., EGFR-mutated nodules tend to be smaller,, and display pleural retraction and speculation. KRAS-mutated nodules are associated with a rounder shape, the presence of nodules in non-tumour lobes and the presence of multiple small nodules. Larger volume multifocal thoracic lymphadenopathies on CT imaging were found to be indicative of ALK mutations.

Figure 1

A schematic representation of different semantic features that can be extracted from medical images.

The different features can be denoted in a feature vector. Feature vectors between the two lesions can differ due to changes in tumour properties.

Semantic features, however, come with their own set of drawbacks and caveats, most chiefly standardization. Different readers can score the same lesion very differently, resulting in a high degree of variability for semantic features. Features derived using such a subjective method may lead the multidisciplinary team to draw incorrect conclusions regarding the diagnosis or treatment response. Moreover, there is a learning curve for readers to be able to generate reliable semantic features, highlighting the risk of inter- and intra-observer variability. Inter- and intra-observer variability is often cited as a point of concern in such radiological studies.32, 33, 34, 35, 36, 37

Another limitation of semantic features is the need for visual assessment by human readers. The tumour description is inherently limited to what is immediately discernable to human eyes, meaning that high-dimensional, and possibly valuable, imaging features will not be taken into account.,38, 39, 40, 41 Early attempts were made to automatically predict qualitative features directly from the image; however, this approach has not seen wide adoption, especially with the rise of fully-automated quantitative features., Although humans excel at pattern recognition, the execution of complex quantitative assessments is often inadequate, making the semantic observations an ideal ‘stepping stone’ for further research to obtain a more quantitative, objective and automatic analysis.,

The quantitative field of radiomics thus presents itself as an automatic and objective method to extract these features from medical images, overcoming many of the shortcomings that plagued semantic features.

Radiomic features

The fundamental concept driving the field of radiomics is that biomedical images contain useful predictive information, beyond what is visible to the naked eye, that can be decoded employing quantitative image analysis., Routine medical images are converted into high-dimensional mineable data, in the form of radiomic features, using mathematical formulae that describe image properties quantitatively.,46, 47, 48, 49, 50, 51 Radiomic feature extraction methods yield a far greater number of features compared with human-derived semantic features, with only a fraction of the effort.

As almost all patients undergo scans for diagnosis or treatment response assessment, a potentially vast database of sources exists for both retrospective and prospective research. From these digital images, a large number of quantitative features can be extracted and analysed for hypothesis generation or experimentation. Radiomic data can be used alone or in combination with semantic features to improve the understanding of underlying biological, therapeutic and post-therapeutic changes, and treatment-related adverse events.

The introduction of radiomics into the imaging analysis workflow adds a promising new tool to better sub-stratify patients and customize treatment plans. Quantitative analysis of images can help to differentiate different types of cancer, predict disease outcomes and determine response to treatment.53, 54, 55, 56, 57, 58

Based on the method of extraction, radiomic features can be divided into two main approaches: handcrafted (or classical) radiomics and deep learning radiomics (see Figure 2).,,

Figure 2

An overview of the two main radiomic pipelines.

The top path shows classical radiomics, with tumour delineation and manual feature engineering and extraction. After feature extraction, statistical models or machine learning can be used to link the features to the endpoint. The lower path shows deep learning radiomics, where the deep learning network extracts features without human interference to link them to the desired endpoint.

Handcrafted features are derived from pre-engineered formulas based on visualized aspects of the image.,, The handcrafted features are often based on intensity histograms, shape attributes and textures, encoding morphological, phenotypic characteristics. Before handcrafted features can be computed, regions of interest (ROIs) within the radiological image must first be delineated (segmented). The combination of delineation followed by handcrafted feature extraction is a defining characteristic of classical radiomics. The extracted features can then be used in either statistical or machine learning models. Delta radiomics, a specific subgroup within classical radiomics, compares (and ultimately subtracts) pre- and post-treatment radiomic features (i.e. feature changes) to generate a new set of ‘delta radiomic’ features meant to reflect treatment-induced morphological changes.,

As a human-dependent process, manual delineation/segmentation is vulnerable to inter- and intra-observer variability. Furthermore, because handcrafted features are extracted mathematically by human-defined and curated quantitative formulas, bias can occur., New AI-driven methodologies have emerged to mitigate many of the perceived pitfalls of classical radiomics.

The second methodological approach to medical image analysis is deep learning, a novel technique that is considered a potential ‘game-changer’ for the field of radiology. Deep learning is a broad term used to describe a collection of neural networks which generate their own features and perform classification without human involvement. It has been shown that deep learning has greater accuracy for specific tasks compared with traditional computer vision techniques.

While an in-depth explanation of deep neural networks is beyond the scope of this review, a small introduction can be given. Many different types of deep learning networks exist, but all are based on the same premise: an artificial neuron. Artificial neurons model our biological neurons and form the basic building block of neural networks. Each neural network contains multiple neurons arranged in different layers. With each neuron obtaining a different value, combinations of these neurons form features such as lines, shapes, patterns and anatomical structures. To obtain the underlying feature extraction and classification parameters, the network needs to learn; this task is commonly referred to as ‘training’. During training, backpropagation, a phenomenon where the neurons are tuned to learn the data, occurs. Backpropagation ensures that the neuron combinations form different features.

In the field of medical image analysis, the convolutional neural network (CNN) is the class of deep learning networks employed most often, as their feature design and extraction are free from human interference. CNNs can generate quantitative features several orders of magnitude greater than the methods applied during semantic feature extraction. Additionally, many deep learning approaches do not require expert delineation/segmentation, further mitigating human bias. With deep learning, feature extraction, selection and classification all occur end-to-end automatically and impartially from human interference.

Radiomic features aim to quantify the morphological phenotype, but they cannot be used for prediction on their own. Rather, the features are combined together, and subsequently, specific predictive radiomic signatures are identified within statistical models/machine learning algorithms. Statistical models, such as univariate or multivariate analysis, are more commonly used in early proof-of-concept studies, where the objective is merely to find mathematical associations between the radiomic features and the relevant clinical endpoint. Machine learning algorithms provide the ability to construct predictive systems that learn directly from the data. Machine learning and deep learning algorithms provide a set of advantages over classic statistical modelling, namely the fact that there are fewer mathematical assumptions and minimal human involvement is required. AI algorithms may be classifiers that try to predict binary/categorical outcomes (KRAS-mutated/wild-type), regression models that give outputs as a continuous number (i.e. body mass index = 15.2), or survival analysis models designed specifically to predict the probability of survival. When designing the AI aspect of a radiomic study, care should be taken when determining which type of machine learning/deep learning algorithm would be optimal for the data and endpoint in question.

Applications of radiomics and AI

Radiomics for prediction of response and adverse events in immunotherapy

Immunotherapy, as an emerging pillar in cancer treatment, has been researched extensively at the basic, translational and clinical levels.66, 67, 68 To date, publications on deep learning/radiomics in the context of immunotherapy are limited. However, as the possibility for synergy between non-invasive imaging markers and immunotherapy gains further recognition, it is anticipated that this will become a major line of medical imaging research.

Broadly classified radiomic research in immunotherapy has focused on either prediction of immune-related adverse events or response, using either RECIST or survival metrics as the endpoint.

With the aim of predicting the immunotherapeutic response, Trebeschi et al. extracted radiomic features from CT images of patients with metastatic NSCLC and melanoma who had received anti-PD1 immune checkpoint blockade. Lesions that displayed a more heterogeneous morphologic imaging phenotype with non-uniform density patterns and compact borders were associated with a better response to immunotherapy. Imaging markers linked to vessel tortuosity were also used to differentiate between responders and non-responders among patients with NSCLC after nivolumab treatment [area under curve (AUC) = 0.79]. Subsequent studies have shown similar associations between specific radiomic features and post-immune checkpoint inhibitor response.71, 72, 73, 74, 75, 76 Tunali et al. demonstrated that combining radiomic features with clinical data enhanced the predictive power of the radiomic algorithm (AUC = 0.80-0.87 with radiomic features and AUC = 0.77-0.78 without radiomic features). A combination of RNA sequence data and CT-derived radiomic features was also able to identify immune infiltration in solid tumours. CT-derived radiomic features combined with delta radiomics was also predictive of an early response to immunotherapy (AUC = 0.857). Overall survival and progression-free survival were also linked to tumour skewness in pre-treatment CT scans of patients with metastatic melanoma treated with pembrolizumab. Magnetic resonance imaging (MRI) radiomic features showed an association with overall survival in patients with melanoma brain metastases receiving immune checkpoint inhibitors.

Some radiomic projects also focused on adverse events seen in cohorts treated with immunotherapy, such as pneumonitis, sarcoid-like granulomatous disease of the lung and, most notably, pseudoprogression.84, 85, 86, 87, 88 Distinguishing pseudo-progression from genuine progressive disease is an essential yet challenging task, often requiring serial imaging to make the distinction. Elshafeey et al. used radiomic features paired with a support vector machine to classify pseudo-progression from progressive disease in patients with glioblastoma (AUC = 0.89). Similar results were obtained by Hu et al., whose work used imaging features derived from eight different MRI sequences (AUC = 0.94). The combination of PET/CT radiomic features with blood parameters was associated with early differentiation of pseudo-progression in patients with metastatic melanoma (AUC = 0.82).

For deep learning, few papers that differentiate pseudo-progression from progressive disease have been published. An intriguing approach is that of Li et al., who combined two novel deep learning techniques (a generative adversarial network and AlexNet) (AUC = 0.92). Other research in the prediction of pseudo-progression with deep learning achieved comparable results (AUC = 0.83 and 0.80).,

Radiogenomics for non-invasive insight into tumour biology for precision medicine

Radiogenomics is the field of research where imaging phenotypes are linked to genetic characteristics (such as gene expression and gene mutation)., The goal is to associate imaging features with previously determined gene expression profiles. In the past years, the term ‘radiogenomics’ has expanded to encompass more than ‘just genomics’. Radiogenomics currently aims to link imaging markers with other biological parameters, such as proteomics, metabolomics and microenvironmental data.

While still a field of research in its infancy, radiogenomics has the potential to add significant value to the future clinical workup of a patient. In an era where targeted therapies are being used increasingly, knowing the genetic profile of the tumour can help in the formulation of a customized treatment plan. Radiogenomics offers the ability to profile the tumour non-invasively and mitigate several challenges facing traditional biopsy-based approaches. In radiogenomics, the full tumour burden could be analysed non-invasively and cost-effectively, accounting for both intra- and inter-lesional heterogeneity., Radiogenomics is also able to analyse patient images at several different time points (e.g. longitudinal imaging of the genomic landscape) and tumour sites; this is not always possible with traditional biopsy-based diagnostic modalities.

Radiogenomics has been applied in many different tumour types. In the field of breast cancer, Yamamoto et al. found correlation between quantitative features extracted from dynamic contrast-enhanced MRI scans and early metastasis and metastasis-free survival biomarkers in breast cancer. Additionally, several studies have used non-invasive imaging features to predict the molecular subtype of the tumour (i.e. luminal A and luminal B). Luminal A subtype breast cancer is less aggressive and has a better response to therapy than luminal B subtype breast cancer. Being able to gain insight into tumour biology non-invasively can impact the treatment plan significantly.

In lung cancer, radiomic research discovered a relation between specific hypoxia markers (vascular endothelial growth factor and EGFR) and quantitative texture features derived from cholangiocarcinoma CT scans. The presence of this imaging phenotype could then be used to indicate a hypoxic microenvironment non-invasively. Similarly, radiomic features have been used to predict the presence of many clinically relevant mutations in NSCLC, namely EGFR,97, 98, 99, 100 KRAS101, 102, 103 and ALK.

Challenges in radiomics

Despite its promising results, the field of radiomics also faces several challenges that need to be overcome.

Generalizability and reproducibility

A requirement for the implementation of biomarkers in the clinic, imaging or otherwise, is that they are independent, informative and reproducible. Imaging biomarkers that meet these criteria are often derived from large heterogeneous datasets and validated on independent test sets. Reproducibility, however, may be hampered due to differences in radiomic values resulting from differences in scanning equipment, image acquisition protocols, reconstruction algorithms,, image pre-processing and feature extraction methods. Essentially, this results in a situation where, based on the healthcare centre and scanning equipment, different radiomic signatures can be found to be prognostic/predictive for the same endpoint. Standardization of scan protocols between different institutions could aid in the generalizability and reproducibility of radiomic features/signatures., Initiatives such as the Quantitative Imaging Network initiative have started to address this concern by putting forward a set of standards and best practices for image acquisition and feature extraction. Lambin et al. have even proposed a radiomic quality score where specific practices within the methodology of an article would be rewarded or penalized. For example, a study with external validation, biological correlation and multiple segmentations per image would receive additional points for its quality score.

Radiomic workflow

The radiomic pipeline can be divided into four main steps: image acquisition, ROI delineation/segmentation, feature extraction/selection, and statistical analysis/AI training. Errors can occur at any point in this pipeline. The build-up of these errors is generally termed ‘propagation of the error’.

As a human-dependent step, ROI delineation/segmentation presents many challenges, most notably inter- and intra-observer variation. Any variation in the radiomic workflow can result in massive changes to the radiomic feature values. Kocak et al. demonstrated that each step in the pipeline was prone to a change of segmentation margin of 2 mm; a margin that can be of high impact and should be taken into account.

Additionally, as expert radiologists are required for accurate delineations, time and availability constraints can often serve as ‘bottlenecks’ for data inclusion in radiomic studies., Automatic AI-driven ROI delineation approaches have therefore become a growing field of research within radiology, but the technology remains in its infancy as it suffers from its own set of caveats.

Future directions

Notwithstanding the challenges mentioned above, the field of radiomics has a bright future, particularly with the research currently being carried out in the following domains.

The rise of deep learning

As indicated in the Introduction, classical radiomics is characterized by a mandatory tumour delineation step, followed by a handcrafted feature extraction step. Delineation is vulnerable to inter- and intra-observer variability as well as human bias, limiting reproducibility. AI methods, particularly deep learning, can mitigate many of the limitations in classical radiomics. Alongside the potential for automatic ROI segmentation, CNNs learn image features internally and link these features to the desired outcomes, all without human involvement. The use of automatic AI methods removes many of the potential sources of human-induced bias from the radiomic pipeline.

Less data-hungry AI

Generalizable deep learning models typically require a large number of well-annotated samples as training data. AI training datasets used by Google, for example, consist of hundreds of thousands of images. One of the significant challenges in medical AI, especially in the case of patients treated with immunotherapy, is the limited amount of publically available high-quality well-annotated medical/imaging data., Current AI approaches, especially deep learning, achieve better predictive performance when trained on large amounts of well-annotated data. Obtaining this high-quality medical data remains challenging, as evidenced by the fact that many of the articles highlighted in this review had low sample sizes (N < 100).

Data augmentation can be used to artificially increase the amount of usable training data to overcome this limitation. Being able to use unlabelled data would unlock even more of the hospital's data for AI development. Towards this, unsupervised neural networks have gained significant interest in medical AI research. In unsupervised learning, the network learns features in the data without having access to the ground truth.

Model explainability techniques: understanding the ‘black box’

Something that raises concern in all of society is ‘fear of the AI uprising’. This sentiment, which is not well founded, reflects the concern that society has with AI. A significant part of this fear stems from the perceived inexplicability of the predictions, regardless of the accuracy (i.e. the ‘black box’). In the context of medicine, clinicians will not accept and adopt technology that seems to behave without any fundamental basis. However, this fear has already existed for more than 30 years, and new methods have been developed to shed light on the inner workings of neural networks.

One approach that has been gaining traction in AI radiological research is saliency mapping. A saliency map is an intensity map that represents the individual importance of each individual pixel/voxel in the generation of the model's prediction. In essence, saliency maps allow for the creation of heat maps that can be overlaid on the anatomical image to indicate which parts of the image were deemed important/relevant by the algorithm for its prediction.

Tumour delineation

Within the radiomic pipeline, tumour segmentation is considered a ‘bottleneck’ as reliable segmentation is time-consuming and requires experienced radiologists. AI has two approaches to mitigate this limitation: automatic segmentation algorithms and non-segmentation-dependent deep learning.

In automatic segmentation, algorithms are created to automatically detect ROIs (based on prior training) and generate delineations/segmentations. Subsequently, features can be extracted from the automatically segmented region. As there is minimal human involvement in this approach, inter- and intra-observer variability can be minimized. One caveat is that in order to create an automatic segmentation algorithm, the AI model must first be trained using human-generated segmentations. It is advised that great care should be paid to the training data to prevent any bias in the data that would lead to a bias in the model. Another mitigation strategy for the segmentation ‘bottleneck’ is the use of CNNs to detect either ‘global’ imaging markers (on the entire non-delineated image instead of an ROI) or have readers generate broad ROIs, termed ‘bounding boxes’. With bounding boxes, the radiologist does not need to delineate the tumour in detail, but can instead indicate the general area of the tumour, for example, in the image.

Multimodal AI models/integrated healthcare systems

Within the routine clinical workflow, patients undergo a wide array of tests and analysis ranging from blood tests to different types of imaging to organ-specific function tests. In essence, from the moment of admission to the moment of discharge, a single patient can generate large volumes of clinical, pathological, radiological and genetic data (among others), all encoding potentially complementary information. AI methods allow for the integration of these different data types to drastically enhance predictive performance. It is envisaged that, in the future, integration systems will be established in hospitals where different diagnostic data are combined (see Figure 3).126, 127, 128 The resulting clinical decision support system will then aid the multidisciplinary tumour boards to design the best possible management plan on an individual patient basis.

Figure 3

A schematic representation of multimodal artificial intelligence (AI) models.

With the increase in available data and knowledge, a combination of different diagnostic data into one integrated diagnostic system can improve AI model outcomes. The outcomes of the integrated diagnostic models can be discussed in multidisciplinary tumour boards to optimize the treatment management plan for patients.

Conclusions

Radiomics is a promising research field with future potential for implementation in the clinical workflow. Whether it is in the context of prognostication, response prediction or tumour biology assessment, radiomic features have been studied extensively with a large number of proof-of-concept studies. Notwithstanding the challenges that need to be addressed, significant progress has been made, especially with the implementation of novel AI methods. It is believed that imaging markers, in general, and radiomics, in particular, will find their place within an integrated diagnostics system that harnesses multimodal patient data.