Research Highlight: Use of Generative Images Created with Artificial Intelligence for Brain Tumor Imaging.

Take-home points

• A generative adversarial network (GAN) is a deep learning technique that enables the generation of new images from unlabeled original images.

• Brain tumor imaging using MRI is an area of particular interest for clinical application of image generation using GAN.

• A recent study demonstrated that the time to progression of glioblastoma can be accurately predicted using synthetic post-contrast MR images.

• GANs can be clinically validated when used as an adjunct to imaging-based deep learning segmentation or classification tasks.

• Further investigations are warranted to determine whether GAN-based synthetic post-contrast MR images may substitute real images.

A GAN is a deep learning technique that enables the generation of new images from unlabeled original images [1]. GANs can learn the data distribution from training samples and generate realistic imaging data that have a similar distribution to the original data but are otherwise different [2345]. Image generation using a GAN is an attractive solution to overcome the limitations of small datasets [67], and the generated images eventually become data inputs and expand the use of deep learning algorithms. Brain tumor imaging using MRI is of particular interest for the clinical application of image generation using GAN, because rare tumor types and the use of multiparametric imaging sequences tend to result in insufficient or incomplete imaging datasets. Although studies have demonstrated the technical feasibility of GANs for creating synthetic images for various purposes, such as filling in missing images [8], cross-modality transfer [9], improving image quality by denoising or creating super-resolution for CT [10], MRI [1112], and PET [13], or segmentation tasks in brain tumors [14], few studies have targeted the clinical implications and evaluated the real-world clinical utility of generative imaging.

A potential clinical use case of image generation using GAN is enabling reduced or no use of gadolinium-based contrast agents (GBCAs) by generating virtual contrast-enhanced T1-weighted images (vc-T1WI) from non-enhanced sequences. Although injection of GBCAs is generally considered a safe procedure, 1.5% of patients have mild adverse reactions [15], and there is general agreement that the safety concerns associated with GBCA should be minimized. The feasibility of creating synthetic vc-T1WI for brain MRI was demonstrated using multi-parametric non-contrast T1WI, T2-weighted images (T2WI), fluid-attenuated inversion recovery (FLAIR) images, diffusion-weighted images, susceptibility-weighted images; diagnostic quality tests and quantitative evaluations were performed [16]. The metrics used to evaluate the quality of GAN-generated images are generally either qualitative (Turing test) or structural similarity indices and the peak signal-to-noise ratio. In addition to the limited number of clinical implementations, there is a lack of evaluation metrics for determining the clinical performance of GANs applied to patient data; thus, they are undertested and not widely applied.

Recently, Jayachandran Preetha et al. [17] investigated the synthesis of post-contrast MRI sequences using pre-contrast MRI sequences, filling in the absence of imaging data for imaging evaluation of glioblastoma (Fig. 1). Their work was unique in that it encompassed both an image-to-image-based task using a GAN and an image-based task using UNet to assess tumor responses in neuro-oncology. Clinical utility was demonstrated by incorporating MRI data from three phase 2 and 3 clinical trials with over 2000 patients. The authors evaluated the clinical performance using an image-based artificial intelligence method for tumor volumetry. Their results showed that prediction of volumetrically-defined time-to-progression was possible with synthetic post-contrast MRI images, and automatic volumetry revealed—on average—no significant difference (0.1 months) between synthetic and true post-contrast MRI sequences. Therefore, very similar performances between synthetic and true post-contrast MRI data in predicting the overall survival were demonstrated.

Fig. 1

Examples of virtual contrast-enhanced T1-weighted images using generative model for glioblastoma, IDH-wild type (left), as compared with real images of glioblastoma, IDH-wild type (right).

IDH = isocitrate dehydrogenase

From this hypothesis-generating study [17], we can obtain ideas about how to apply and validate GANs in clinical cases. First, synthetic images can be used as direct substitutes for real images and can make the use contrast media for MRI or CT imaging or an additional radiation exposure for X-ray, CT, or PET imaging optional, thereby reducing the harm or cost associated with the extra imaging procedures. The utility of a GAN can be demonstrated by generating synthetic data to fill in the absent or insufficient data in a multicenter trial [18]. Second, because evaluation metrics applied to images are often based on technical similarity or image quality itself, the clinical performance of synthetic images can be measured to fulfill further image-based tasks of detection, segmentation, and classification, which are often used as evaluation metrics to determine the clinical utility of imaging (Fig. 2). Currently, the evaluation metrics applied to GANs largely focus on the image quality and diversity of the generated images [19], often without a clear reference standard [20]. The diversity of the generated images was clinically shown in a study on the radiologic features of molecular subtypes of gliomas on MRI [21]. Likewise, when used as an adjunct to imaging-based tasks of segmentation or classification, the clinical performance of generative images and models can be measured, and it can be determined whether they would have a clinical impact by enhancing datasets, reducing harm, or increasing benefit.

Fig. 2

Diagram demonstrating how generative imaging can be used and validated in a clinical workflow.

Generative images can be applied during the data input stage and may improve prediction performance during every process of artificial intelligence in neuro-oncologic imaging, including detection, segmentation, and subsequent classification. FLAIR = fluid-attenuated inversion recovery

However, whether virtual GBCA enhancement can replace contrast-enhanced T1WI in neuro-oncology remains to be addressed. The imaging requisites for synthetic vc-T1WI were multi-parametric imaging, including T1WI, T2WI, FLAIR [17], and additional diffusion-weighted and susceptibility-weighted sequences [16]. A feasibility study showed that diffusion-weighted imaging and T2WI contributed to demonstrating peritumoral edema, cellularity, and necrosis [16], while Jayachandran Preetha et al. [17] found that T2WI and FLAIR contributions were larger than diffusion-weighted imaging because the contrast enhancement of brain tumors is based on disruption of the blood-brain barrier, not on cellularity. The different prerequisites for imaging sequences show that the technique is yet to be optimized and further studies are warranted. As a clinical task, glioblastoma is a large tumor for which tumor segmentation produces a reliable measurement, but the identification of comparatively subtle and small contrast enhancements, for example, in brain metastasis and lower-grade gliomas (or active lesions in multiple sclerosis), is far more challenging because the available information in pre-contrast MRI sequences for this task might be insufficient and because synthetic images are limited (distorted) with respect to small vessel structures and image smoothness, rather than the generation of large enhancing (tumor) regions. One potential strategy to—at least partially—address these limitations could be the use of low-dose GBCA administration schemes (e.g., with 10% of the full-dose) and use GAN-based approaches to synthesize (virtual) full-dose contrast-enhanced T1WI [22]. Finally, the benefit of GBCA excels the potential adverse effects in glioblastoma, especially in the differentiation of pseudoprogression from true progression when brain MRI is crucial for an early diagnosis and accurate diagnosis needs to be pursued rather than reducing GBCA because the diagnosis substantially impacts patient treatment.

In summary, by generating post-contrast enhancement images as an adjunct to deep learning segmentation, GANs can be useful for the quantitative measurement of surrogate endpoints of tumor progression. As there is no clear reference standard for measuring the clinical performance of GAN, adjunctive imaging-based tasks of deep learning segmentation or classification will help measure the clinical performance of GAN. Image generation using GAN may potentially add substantial value by reducing the risks associated with imaging, including the use of contrast agents or radiation exposure.