Generative Adversarial Networks in Digital Pathology and Histopathological Image Processing: A Review.

Digital pathology is gaining prominence among the researchers with developments in advanced imaging modalities and new technologies. Generative adversarial networks (GANs) are a recent development in the field of artificial intelligence and since their inception, have boosted considerable interest in digital pathology. GANs and their extensions have opened several ways to tackle many challenging histopathological image processing problems such as color normalization, virtual staining, ink removal, image enhancement, automatic feature extraction, segmentation of nuclei, domain adaptation and data augmentation. This paper reviews recent advances in histopathological image processing using GANs with special emphasis on the future perspectives related to the use of such a technique. The papers included in this review were retrieved by conducting a keyword search on Google Scholar and manually selecting the papers on the subject of H&E stained digital pathology images for histopathological image processing. In the first part, we describe recent literature that use GANs in various image preprocessing tasks such as stain normalization, virtual staining, image enhancement, ink removal, and data augmentation. In the second part, we describe literature that use GANs for image analysis, such as nuclei detection, segmentation, and feature extraction. This review illustrates the role of GANs in digital pathology with the objective to trigger new research on the application of generative models in future research in digital pathology informatics. Copyright:

INTRODUCTION

Digital pathology is rapidly gaining prominence with the development of modern imaging modalities and advancements in novel technologies. During histopathology assessment, which is the current gold standard for cancer diagnosis, surgical samples obtained from tumor biopsy and/or resection are typically prepared and stained using hematoxylin and eosin (H&E) stains and/or other techniques.[1] Hematoxylin stains cell nuclei blue/purple in color, whereas eosin gives a pink color to the cytoplasm and the connective tissues. Manual examination of the H&E stained slides is performed by trained pathologists for tumor diagnosis and grading, with the aim to guide further treatment and to give prognostic information. The inception and evolution of whole-slide imaging in the late 1990s has enabled complete histopathological slides to be recorded and digitally stored at high resolution using microscope-based slide scanners or whole-slide scanners.[2] A whole-slide image (WSI), also referred to as virtual slide, is the digital representation of a histopathology slide which provides visual information about the H&E and/or other staining techniques, in a range of multiple scales and different focal planes.

The increasing presence of digital pathology in contemporary medicine has motivated many researchers to explore the opportunities in digitally analyzing histopathological images for automated pathological diagnosis and decision-making.[34] Several novel methods in image processing, including computational fractal-based analysis and artificial intelligence (AI), have been tested against confirmed cases in their ability to extract reliable information from images to extend the limits of human interpretation beyond viewing a microscopic slide.[56] Several studies have applied fractal dimension analysis for the diagnosis, staging, and prognosis of different cancer types.[78] The application of AI methodologies in digital pathology has enabled pathologists to maximize the knowledge extracted from WSIs and has facilitated the automation of tasks that are traditionally manual and operator dependent. Niazi et al.[9] reviewed the role of AI in digital pathology by highlighting its applications, including education (training next generation of pathologists) and quality assurance (assessing interobserver and intraobserver variance). The synergetic integration of pathology with AI could lead to improvements in education and quality in the health care and medical field.

Deep learning is emerging as a powerful tool in machine vision and natural language processing and is opening new venues in AI with transformative impact on medicine, including neurosciences and digital pathology.[1011] While traditional methods usually require “handcrafted” domain-specific features, deep learning methods can directly and automatically detect features from the data without the need for labour intensive manual feature extraction. Deep learning methods are used in digital pathology in many tasks including image preprocessing,[12] segmentation and detection of histologic primitives,[1314] and grading and prognosis.[1516] In 2014, Goodfellow et al.[17] introduced the generative adversarial networks (GANs) as a deep learning concept capable of learning representative distributions of data and generating new, synthetic data that can be used as real data substitutes or complements.

GANs have achieved state-of-the-art performance in many image generation tasks,[18] including text-to-image synthesis,[19] super-resolution images,[20] unpaired image-to-image translation,[21] and various applications in medical imaging such as image reconstruction,[2223] image synthesis,[24] segmentation,[25] classification,[26] and other tasks.[2728] Conditional GANs (cGANs), a variant of GANs, have been able to virtually stainimages of unstained specimens, to make them resemble H&E stained pathology slides.[29] This research using the GAN model has shown the possibility that digital virtual staining technology could lead to replacement of the laborious and expensive process of actual histological staining.

The aim of this paper is to provide an overview of the role of GANs in digital pathology, thereby triggering new research on the application of generative models in future research in digital pathology informatics.

METHODS

Search strategy

The papers used in this review were retrieved by conducting a search on Google Scholar. Meanwhile, as GANs are relatively new, a substantial number of articles are still in the publication process toward different journals and conferences, so we have also covered pre-prints published in arXiv. The search was limited to the English language and a year range of 2014–March 2021 using the keywords “histopathol*” and “generative adversarial network*”, “histopathol*” and “gan”, “histology” and “generative adversarial network*,” “histology” and “gan.”

Inclusion criteria

We included papers that made use ofH&E stained digital pathology images for histopathological image processing. All papers that (i) applied GANs to other imaging modalities such as ultrasound imaging, magnetic resonance imaging (MRI), computed tomography, and positron emission tomography or (ii) used digital slides which were stained with staining techniques other than hematoxylin and eosin were eliminated. Out of the 435 results listed from the search query, 48 articles were chosen for the study after applying a filter of title and abstract on the retrieved results to eliminate irrelevant papers. Figure 1 shows how the selected publications are distributed over different areas of image processing in digital histopathology.

Figure 1

The distribution of papers selected for this review between the different areas in histopathological image processing that use generative adversarial networks, dated between 2014 and March 2021. Others here refer to tasks such as domain adaptation, image synthesis, image enhancement, and virtual staining

OVERVIEW OF GENERATIVE ADVERSARIAL NETWORKS

In this section, we provide a general description of GANs and a few extensions such as cGANs and cycle-consistent GANs (cycleGANs), which are used as the basis for further model development in histopathological image processing.

Generative adversarial network

GANs are a special type of neural network model consisting of two networks: a generator network and a discriminator network which are trained simultaneously when competing with each other.[17] The generative network function G (x) generates new data and the discriminative network function D (y) learns to distinguish between the real and the generated data. Both networks always compete to optimize themselves; the discriminator D tries to maximize its objective function and the generator G tries to minimize its objective function. Equation 1 mathematically expresses the objectives of D and G as:

where D (x) is the discriminator, G (z) is the generator, x is the real data, and pz (z) is the input noise. Both networks train each other through multiple cycles of generation and discrimination when simultaneously trying to outwit each other. This is treated as a two-player minimax game which converges at a state known as the Nash equilibrium.[1730] Figure 2a shows a graphical representation of the GAN model.

Figure 2

Graphical representation of (a) the generative adversarial network model, (b) the conditional generative adversarial network model, and (c) the cycle-consistent generative adversarial network model. For more details please see references 17, 21 and 31.

GANs have gained attention in many areas due to their effectiveness in generating new synthetic images which can complement or even replace real data.[323334] GANs can be potentially utilized in several steps of digital pathology, including synthesizing images for augmenting the training data and processing of histopathology images. State-of-the-art GANs, such as a medical imaging GAN developed for retinal images, have recently shown impressive results in generating synthetic images and their segmented masks which can be used for the application of supervised analysis of medical images.[32] Moreover, GAN models have been proposed as useful tools to learn complex representations of cancer tissues in histopathology images.[2835]

Conditional generative adversarial networks

cGAN are an extension of the GAN architecture, in which both the generator and discriminator are conditioned on auxiliary information such as a label associated with an image or a more detailed tag.[31] The condition information c is fed into both the generator and the discriminator as an additional input to direct the data generation process. The generator learns to generate a sample with this specific condition or characteristics rather than a generic sample from unknown noise distribution. Equation 2 mathematically represents the loss function of cGAN as:

where D (xǀc) and G (zǀc) demonstrate discriminating and generating an image given a condition c, x is again the real data, and pz (z) is the input noise. Figure 2b shows a graphical representation of the cGAN model.

Cycle-consistent generative adversarial network

A CycleGAN is another GANs' variant, capable of translating images from one domain to another, for example, translating images of zebras to images of horses.[21] Image-to-image translation is traditionally performed by mapping between an input image and an output image using a training set of aligned image pairs. However, for many tasks, paired training data will not be available. The cycleGAN is capable of capturing special characteristics of one image domain and translating the image into another domain, without having paired images. The model of cycleGAN includes two generator networks G and F with mapping G : X → Y, F : Y → X and two discriminator networks D (x) and D (y). While D (x) aims to distinguish between images in domain X and translated images {F (y)}, D (y) aims to discriminate between images in domain Y and {G (x)}. Figure 2c shows a graphical representation of the cycleGAN model.

GENERATIVE ADVERSARIAL NETWORKS IN HISTOPATHOLOGICAL IMAGE PREPROCESSING

Histopathological WSIs are large-sized images that provide a great source of information. Nevertheless, the information contained in the histopathological WSIs is affected by many factors such as the type and quality of the microscope or scanner used for imaging, the size and magnification of the image, vibrations caused by external factors, physical color variation due to staining and tissue section preparation process, contamination in the sample, noise and defocusing, and artifacts. Consequently, preprocessing of histopathological WSIs is essential to improve the quality of the images prior to applying AI learning algorithms. The importance of preprocessing is illustrated by a recent study where various preprocessing techniques have been applied to histopathological WSIs before feeding them to a convolutional neural network (CNN) architecture for classification.[36] The results showed an improved classification performance with traditional preprocessing techniques such as background noise reduction and cell enhancement, as compared to the original images. Excessive additional preprocessing, however, for example, thresholding or the application of morphological operations applied on the already preprocessed images, resulted in removal of the important image features and subsequent performance reduction of the CNN.

The introduction of GANs has triggered a novel approach for histopathological image preprocessing. CycleGANs have been effectively applied to color normalization of breast histopathological images eliminating the requirement of having a reference template slide.[37] Super-resolution GANs (SRGANs) are a variant of GANs that are capable of inferring photorealistic natural images. SRGAN has been employed to increase image resolution and eliminate image noise from breast histopathological images.[38] The SRGAN-based method produced high-resolution images that are cleaner than images preprocessed by classical interpolation methods for increasing resolution. The following subsections describe various preprocessing tasks using generative models: color normalization, virtual staining of histological tissue, image enhancement, removal of ink marks from histopathological images, and data augmentation.[27394041516667]

Color normalization

Color variations in digital slides hinder the successful performance of deep learning algorithms in digital pathology across laboratories and settings. Physical color variations can be a result of the biochemical staining process (difference in stain batches, variability in staining amount, time, thickness of sections, and staining protocol), while color variations can also be due to the imaging and digitizing parameters (variations in illumination, (de) focusing, imaging resolution and magnification, microscope or scanner model, and spectral sensitivity of the detector).[4243] Normalization of the color represented by digital slides is thus an important preprocessing task in digital pathology. An example of observed color variations in the sample images is shown in Figure 3.

Figure 3

Original H&E stained glioblastoma pathology slides obtained from The Cancer Genome Atlas database[44] showing diverse color variations in the sample images

Traditional image processing approaches such as histogram techniques[4546] and color models[4748] have been used for color normalization for a long time. The progress in artificial neural networks such as convolutional neural networks and the generative models has opened innovative paths for color normalization. Some of these GAN-based techniques and approaches for color normalization in digital pathology are highlighted in Table 1. The listed color normalization approaches are based on a style transfer method in which the style of the input image is modified based on the style image, when preserving the content of the input image.[37395051535455] The methods based on cycleGAN explore the capability of unpaired image-to-image translation which makes it a flexible architecture for stain normalization. Other approaches discussed here use alternative formulations such as self-attention models,[56] cGAN,[31] and encoder–decoder architecture.[57] The stain normalization stage can be integrated into a classification approach to eliminate bias introduced in these types of architectures.[5058]

Table 1

Novel generative adversarial networks approaches used for color normalization of histopathological images

Reference	Tissue type	Dataset	Architecture	Method
Cho et al., 2017[49]	Lymph node samples	CAMELYON16	cGAN	Stain style transfer approach
Bentaieb and Hamarneh, 2018[50]	Breast histology images, colon adenocarcinoma tissue images, ovarian carcinoma images	MITOS-ATYPIA14 challenge, MICCAI16 GlaS challenge, nonpublic dataset	GAN	Stain style transfer approach using encoder-decoder architecture and skip connections
Zanjani et al., 2018[39]	Lymph node samples	Nonpublic dataset	GAN	Unsupervised GAN based model for stain color normalization
Rana et al., 2018[51]	Prostate core biopsy tissue samples	Nonpublic dataset	cGAN	Staining and de-staining models used for learning hierarchical non-linear mappings between non-stained and stained WSIs
Zhou et al., 2019[52]	Breast cancer samples	CAMELYON16	cycleGAN	Enhanced cycleGAN based method using stain color matrices for translation
Shaban et al., 2019[37]	Breast cancer samples	MITOS-ATYPIA14 challenge, CAMELYON16	cycleGAN	Structure preserving stain style transfer
Cai et al., 2019[53]	Breast cancer samples	MITOS-ATYPIA14 challenge	cycleGAN	Structure preserving stain style transfer
Shrivastava et al., 2019[54]	Duodenal biopsy samples	Nonpublic dataset	GAN	Stain transfer approach using self-attentive adversarial network
Salehi and Chalechale et al., 2020[55]	Breast cancer samples	MITOS-ATYPIA14 challenge	cGAN	Pix2Pix based stain to stain translation

cGAN: Conditional generative adversarial networks, GAN: Generative adversarial networks, WSIs: Whole-slide image

Virtual staining of histological tissue

GANs have gained significant attention in many other medical image processing tasks, including virtual staining of tissues slides.[2729515960] Histological staining is commonly used in pathological diagnosis to highlight the important features of tissue. These staining processes are usually lengthy, relatively costly, and laborious due to time-sensitive steps.

Virtual staining can be achieved starting from unstained slides imaged using different imaging modalities. In one study, lung tissue slides were imaged using a hyperspectral microscope where the tissue structure is imaged at many different wavelength bands (colors) of the light spectrum.[29] The most important information from this multidimensional image cube was captured by principal component analysis to reduce the cube to a three-component image which was subsequently fed to a cGAN to train the staining network model. The trained network was able to transform images from the hyperspectral domain to the H&E domain by highly nonlinear mappings, resulting in their virtually stained versions.

Virtual staining can also be achieved using conventional broadband light illumination, by training a cGAN using RGB-colored WSI pairs before and after H&E staining.[51] In addition, a “destaining” model was developed that learned the reversed mapping between an RGB image pair of an H&E stained WSI and a non-stained WSI of the same biopsy. The authors could examine their models qualitatively by comparing the original H&E stained section to their virtually destained and restained counterparts; however, further suitability of the generated images for tumor diagnosis was not pursued and further clinical validation is necessary.

A related domain transfer approach in virtual staining techniques is the transfer from H&E stained specimens to their immunohistochemistry (IHC)-stained equivalents. IHC stains target specific proteins on cells on the tissue slide and display different colors. A conditional cycleGAN was trained to achieve unpaired image-to-image translation for multiclass virtual IHC staining.[27] Optimized structure and authenticity preservation was achieved by including additional loss functions for structural and photorealistic properties of the created virtual slide.[6162] To evaluate the results, computer vision researchers were asked to score the image quality of real and fake images, and professional pathologists were asked to score the staining quality of the virtually stained patches. Although the conditional cycleGAN had a higher authenticity score as compared to the cycleGAN, both models suffered considerably from incorrect staining. The authors suggest to improve this domain transfer issue by increasing and optimizing the training dataset.

The ability of virtually staining is also demonstrated in many other image modalities such as autofluorescence image captured by a standard fluorescence microscope,[60] quantitative phase images,[59] confocal microscopy images,[63] and holographic virtual staining of individual biological cells.[64] Virtual staining of histological tissues has the potential to bypass the disadvantages of manual histological staining, provided it is clinically validated and approved. In addition, virtual staining preserves the unlabeled and unaltered tissue sections for further analysis.

Image enhancement

Image enhancement acts as a preprocessing step that improves the quality of the images in terms of color, brightness, and contrast. Researchers in this field have explored several image enhancement techniques using AI methods.[65] Deep neural networks are employed for image quality enhancement but need a large quantity of paired training data of low-quality images and their corresponding high-quality images. To alleviate this barrier, SRGANs can generate high-resolution images, and for such a reason, they have attracted attention among the researchers in this area. For instance, SRGANs have been employed to generate super-resolution images and to eliminate noise from breast histopathological images.[38] Another study proposed a mixed-supervision GAN, based on SRGAN, that can take both high- and medium-quality images as input for training.[66] This approach leverages training data from high quality to medium quality to improve the performance when limiting the costs of data curation.[66] The method uses a sequence of generators and discriminators, dealing with increasingly higher quality training data at each level, compared to the low-quality input data. Quantitative evaluation of the images was done using metrics such as the structural similarity index and the relative root mean square error, while the images were not tested for further image analysis such as classification and segmentation tasks.

Another study employing SRGAN performed classification of super-resolution, high-resolution, and low-resolution images along with the comparison of different performance measures.[67] This study uses a wide attention SRGAN which uses wide-residual blocks and a self-attention layer. Wider channels before activation in residual blocks were shown to improve the performance of image super-resolution networks.[54] By including an in-built self-attention function that calculates the weighted mean of all pixels in an image, the model learns relationships between widely separated spatial regions and distant pixels based on patch appearance similarity. The classification results for high- and low-resolution images displayed an accuracy of 99.49% and 95.82%, respectively.[67] These studies emphasize the role of the ability of GANs to increase the resolution during the reconstruction of poor-quality images, thereby tackling the cost of hardware and storage to employ high-resolution images.

Ink mark removal

Histopathological glass slides are often annotated by pathologists using pens or other markers to indicate regions of interest, such as neoplastic areas or other specific features. Figure 4 displays some examples of ink-marked WSIs from The Cancer Genome Atlas dataset.

Figure 4

Examples of whole-slide images annotated by (a) green and (b) blue ink marks obtained from The Cancer Genome Atlas database[44]

This pragmatic approach unfortunately hinders computer-aided interpretation of marked WSIs. For example, color thresholding or other automatic selection algorithms fail to select the correct areas in the presence of ink marks. State-of-the-art color normalization methods are not sufficient in removing ink marks as they are designed to handle sensitive variations in unmarked H&E images;[68] therefore, other solutions based on generative models have been explored.

A fully automatic CNN-based approach to remove ink from H&E stained prostate WSIs consisted of three separate architectures.[40] First, a sequential classical CNN classifies the image tiles into three categories: ink-free tiles, background contaminated tiles (tiles with ink marks, but not on the tissue), and foreground contaminated tiles (tiles with ink marks on the tissue). Subsequently, the background contaminated tiles are cleaned by replacing the ink marks with a white background. A fast region detector performs the precise localization of the ink marks on the foreground contaminated tiles. Finally, a cycleGAN is applied to these detected localized marks to restore these image tiles to ink-free image tiles.

A similar study has approached the ink removal as a style transfer problem using a cycleGAN to remove ink marks from human melanoma tissues, when preserving the tissue structure underneath the marker region.[69] The melanoma WSIs were first tiled to obtain marker patches which contain full or partial ink marks and clean patches which do not contain any ink marks. The cycleGAN used, here, is composed of two generative and one discriminative CNN. The first generative CNN is trained to remove ink marks from the marker patches and to generate patches similar to clean tissue that cannot be differentiated by the discriminator model. The second generative CNN is trained to reconstruct the output of the first generative CNN to ensure the preservation of the individual visual information from each image patch. The application of cycleGAN to ink removal exploits the image-to-image translation method to produce visually coherent images and outperforms the state-of-the-art color normalization methods that are not capable of removing ink marks from the WSIs.

Data augmentation

The availability of large training sets is essential to achieve good performance in deep learning. However, due to privacy concerns, high cost, and complexity of data collection and labeling, medical datasets, especially pathological datasets, are relatively small. Data augmentation is a method to significantly increase the amount and variety of data available for training models, without collecting new data. Verified data augmentation methods can be used in the development of automated medical image analysis to mitigate the effects of data imbalance and improve overall performance of the training model.[7071] Traditional data augmentation methods include image transformations (rotation, flipping, reflection, zooming, etc.) and color transformations (histogram matching). The success of data augmentation methods has stimulated an interest in researching advanced approaches to generate new images for training. After the initial development of GANs, generative models that synthesize completely new images have become a popular method for generating augmented data.

GAN-based data augmentation methods were able to generate new skin melanoma photographs, histopathological images, and breast MRI scans.[41] Here, the GAN style transfer method was applied to combine an original picture with other image styles to obtain a multitude of pictures with a variety in appearance. The augmented dataset was utilized for diagnostic image classification. An image translation approach using cycleGAN was used in another study to generate synthetic colorectal polyp images from normal colonic mucosa images.[72] The generated synthetic images maintained the general structure of the normal images but exhibited adenomatous features. The quality of the augmented images was then evaluated by a pretrained classifier and through clinical evaluation by pathologists, finding that three of the four pathologists could not differentiate at least half of the synthetic images from real images. In another study, cGAN was used to synthesize realistic cervical histopathology images to expand a limited training dataset.[72] The authors used a filtering mechanism to control the feature quality of augmented synthetic images and boost the performance of the classifier. A cycleGAN was proposed to augment existing nuclei segmentation datasets by generating synthetic H&E patches representing several different organs for nuclei segmentation algorithm development.[33] Another study has recently investigated the use of GAN to augment the dataset of histological specimens of gliomas with synthetic gliomas, to increase the training and accuracy of deep learning techniques to predict the IDH gene mutation status.[73] The accuracy of the deep learning model for predicting the IDH status onH&E stained histopathological images was increased from 80% (without augmentation) to 85% with GAN augmented data. Figure 5 shows some examples of GAN-synthesized glioma images.

Figure 5

Samples of generative adversarial network synthesized glioma images from coarse to fine scales

The studies listed in this section show the capability of GANs to generate synthetic images: from image-to-image translation for normalization and cleaning of color-shaded and marked pathology images to the generation of augmented databases to solve histopathological data limitations. Further exploration of the application of GANs to WSI preprocessing will help in the workflow of digital pathology as it preserves the tissue sample structure when enabling further AI-based image analysis. These studies also show the capability of GANs to generate synthetic images that can alleviate some of the limitations related to histopathological data.

GENERATIVE ADVERSARIAL NETWORKS IN HISTOPATHOLOGICAL IMAGE ANALYSIS

Computer-aided analysis of images has been widely explored in many areas of studies for feature extraction and pattern recognition. With the recent advancement of digital pathology, histopathological image analysis has gained attention of many researchers.[74] Computer-assisted quantitative analysis of histopathological images can significantly speed up clinical analysis and research project outcomes. In addition, AI has the potential to uncover information in pathology images that results in novel insights about diseases and therapy. GANs and their extensions have opened encouraging ways to challenge long-standing image analysis problems such as segmentation, feature extraction, detection, and classification. In the following subsections, we categorized the histopathological image analysis articles into: automated detection and segmentation, other feature selection, extraction and quantification, and domain adaptation.[287576]

Nuclei detection and segmentation

Automated segmentation and detection of tissue and cellular structures is an important task in histopathological image analysis. Classical methods for nuclei segmentation techniques include watershed algorithms, color-based thresholding, morphological processing, active contours, and their variants.[77] During the last few decades, numerous state-of-the-art methods have been proposed for nucleus detection and segmentation in digital pathology images.[78] GANs have also effectively enabled a range of nuclei segmentation tasks, including segmentation of nuclei from WSIs and the identification of nuclei within a specific region.

In an unsupervised nuclei segmentation method formulated as an image-to-image translation problem, diverse histopathological images were selected as nuclear domain and randomly generated nuclear shapes were selected as label domain.[75] Then, a cycleGAN was used to translate the histopathology images to the label domain by generating segmentation maps. Several challenges were reported in this study, including (1) reconstruction losses for segmentation resulting in the suppression of less bright nuclei with each gradient update, and (2)incorrectly generated nuclei shape annotations resulting in poorly segmented nuclei. Despite the challenges, the study showed that generative models can be a promising solution for the nuclei segmentation problem, including the separation of differently shaped nuclei. In another study, nuclei detection on colorectal adenocarcinomas histopathology images was peformed using a residual attention GAN, which is based on cGAN.[79] The residual attention mechanism is used to capture the characteristics of the nuclear regions more clearly by generating a more accurate probability map with spatial contiguity, resulting in improved nuclei detection.

The performance of deep learning algorithms for segmentation highly depends on the quality and volume of labeled histopathology data available for training. Furthermore, most of the work has focused on developing cellular and cellular feature segmentation methods for single organs. In a multiorgan study, cGANs are used to train synthetically generated data and real data to perform nuclei segmentation.[80] First, a cycleGAN model was trained with images from four different organs to synthetically generate pathology data with perfect nuclei segmentation labels. These real and synthetically generated images were subsequently used to train a cGAN to perform nuclei segmentation. The study demonstrates that this approach performs better than the state-of-the-art methods for nuclei segmentation.

The concept of the information-maximizing GAN (infoGAN) architecture,[35] which learns the visual representations of the cell nuclei by utilizing features in the image such as shape, nuclear density, and color, was explored for the classification of breast cancer histology images.[81] First, a CNN-based stain transfer method is used to normalize the H&E stained breast cancer histology images. The stain-normalized breast histology images are then trained by a neural network to segment out the nuclei. Finally, infoGAN is used to classify the segmented nuclei in an unsupervised manner based on visual representations of different types of nuclei.

Feature selection, extraction, and quantification

Selection, extraction, and quantification of relevant features from histopathological images have long been an active area of research within the image processing community. Automated cancer diagnosis relies on capturing cellular and tissue level features to quantify the changes in a given tissue.[45] Traditional methods used for feature extraction include morphological, topological, textural, fractal, and/or intensity-based methods.

Pathology GAN[28] is a framework that generates high-fidelity synthetic cancer images using GANs to capture the key tissue features such as color, texture, spatial features of cancer and normal cells, and their interaction. The proposed model uses the generated images to obtain the feature space based on cell density and morphology in cancer tissue and captures pathologically meaningful representations. The study also demonstrated that pathologists were not able to reliably find differences between the real and generated images. Another study used a GAN architecture to perform unsupervised representation learning for cell-level images segmented from bone marrow histopathology images.[35] Each cell-level image contains only one cell, enabling GAN-based learning of interpretable and disentangled cellular elements. These cell-level visual representations can be used for a variety of tasks such as cell classification based on their semantic features. The proposed pipeline used the cell-level information to calculate the cellular class proportions in the histopathological images, combining this with nuclei segmentation to perform image-level classification of histopathology images. Then, image-level classification was performed to differentiate between normal and abnormal bone marrow pathology images. The study revealed that the algorithm can classify different cell types such as myeloblasts, monocytes, granulocytes, lymphocytes, and erythrocyte precursors and quantify their presence to confidently discriminate normal from abnormal images. However, some abnormal images were wrongly classified as normal which the authors attribute to the atypical staining of erythroid precursors. The authors suggested improvements to make the model more robust to these variations and also highlighted that acquiring high-quality histopathological images is a priority for reliable results.

To improve segmentation accuracy, a GAN-based image-to-image translation method was proposed that is based on image content enrichment rather than increasing the number of samples, by overlapping virtual images with the given (original) image.[82] First, the input image to be segmented is translated to different domains (stains) by several generators to produce virtual images with stains different from the original image. All these virtual images are concatenated and merged with the original image, resulting in an enriched multichannel image. As the second step, the enriched images are used to train and test a fully connected convolutional network for segmentation. An experiment with mouse kidney pathology images showed an improved segmentation score for kidney glomeruli for enriched images compared to using the original images only. The study relies on the rationale that the two models individually trained to solve a specific subproblem are more powerful than one single model trained to solve the whole, more complex problem. The image-to-image translation approach here produces highly realistic and detailed histopathological images with added information from the virtual stains, leading to improved segmentation.

Domain adaptation

Domain variations such as differences in staining between laboratories can prevent the predictions based on a model learned from one domain being directly applicable to another domain (other laboratory). Oftentimes, paired images from the different domains (for example, the same samples imaged at different laboratories) are not available; thus, direct transformation cannot be learned. Domain adaptation is the task of transferring the image features between domains, so that classification based on the annotated source domain can be reliably applied to the unlabeled target domain. A study was proposed for classification and grading of breast cancer images by transforming the target images to the source domain and then applying a deep learning method.[76] The authors used cycleGAN for unsupervised domain adaptation to transform data from four different medical centers to data from one center to reduce the variability in the data and improve the classification results. Another study[26] addressed unsupervised domain adaptation using a different paradigm, namely the alignment of the learned image feature space of the source domain with the unlabeled target domain. The adversarial training minimized the distribution discrepancy in the feature space between the source and target domains. Asymmetric adaptation resulted in fine-tuning of the target domain network guided by the source-domain network, thereby mimicking the distribution of the source-domain feature space. Thus, the target network gets trained to extract the domain invariant features from input samples. Using a set of prostate histopathology images, the classification results showed improvement compared to baseline models, indicating that adversarial training may not only decrease color differences but also mechanical distortions, morphologic, and structural differences resulting from different processing routines.

In medical image processing, learning of distinctive patterns and feature extraction plays an important role in the diagnosis of diseases. The discussions in the above sections show the rich feature extraction capability of GANs, their power to extract semantically meaningful information from the histopathological images, and transferring the image features between different domains.

DISCUSSION AND CONCLUSION

Digital pathology is rapidly growing alongside the recent developments in AI and deep learning. The integration of AI with pathology can contribute exciting changes to health care, including precision medicine through personalized patient models and treatments.[83] In particular, the innovative development of GANs and its variants has led to an improved digital pathology workflow resulting in more informed and comprehensive cancer diagnosis.[2882] In recent years, GANs have gained significant attention in the area of various medical image processing tasks, including virtual staining of tissues slides, which bypasses the laborious and expensive process of histological staining.[5960] We consider the application of GANs to digital pathology as a promising upcoming research area with many opportunities to be explored. Fortunately, it is gaining the interest of the computational and medical researchers.

In this study, we have presented the application of GAN models to histopathological image processing. The above discussion shows that GAN has the power to facilitate several practical applications in digital pathology. The advantages of GANs in digital pathology can be summarized as the following.

Resolving scarcity of training datasets by producing realistic synthetic data

Due to privacy concerns, high cost, and complexity of data collection and labeling, medical datasets tend to be relatively small compared to the general datasets. GAN-based models can be a promising solution for the limited size of publicly available clinical datasets. The ability of GANs to mimic data distributions and produce realistic synthetic data can be employed to augment data, enlarge the medical datasets, and handle class imbalance. Furthermore, the manual labeling of WSIs is a time-consuming task that could be solved using synthetic labeled images.

Augmentation by generating data with diversity

Augmentation of data can be performed in many ways, such as image transformation and color transformation methods. However, these augmentation techniques do not provide diversity in the data being generated. GANs are additionally effective for the augmentation that is difficult to implement with standard augmentation techniques, through its capability to generate realistic and customized variations.

Preprocessing of input data

GANs have proved to be effective candidates for image preprocessing tasks, such as color normalization, ink removal, and data augmentation. We envisage that it could be further explored to use generative models for the development of universal image preprocessing steps that can be applied to all types of histopathology images irrespective of the sample types.

Virtual staining of the whole-slide images

Histological staining of tissue slides is usually a laborious and expensive process during pathological diagnosis. GANs can be a potential candidate for virtual staining of tissue slides which can generate images with reduced stain variations. This will eventually reduce the labour and cost incurred in the actual histological staining.

Effective in learning and extracting features

Feature extraction and pattern recognition play an important role in medical image analysis for the diagnosis of diseases. GANs are powerful tools in extracting and quantifying semantically meaningful information from images, with several advantages in digital pathology for automatic detection, segmentation, and classification applications.

Although the number of studies applying GANs has increased significantly in the last 3 years, it should be noted that most of the research has been focused to MRI instead of other medical imaging modalities. One reason for this might be the large number of publicly available MRI datasets compared to the pathological datasets. In the area of histopathology, most of the GAN-based methods applied to histopathological images focused on breast cancer or prostate cancer datasets. This leaves ample scope to adopt these state-of-the-art GAN-based methods to datasets representing other organs.

Despite its significant success as a generative model, the training of GAN is an expert task and the training process encounters many challenges such as vanishing gradients, mode collapse, and failure to converge. The possible solutions to handle GANs challenges are to choose appropriate network architectures, to use customized loss functions, and to apply various forms of regularization.[84] Another important challenge with GANs is that they are often focused on matching the input image to a target distribution of images, sometimes resulting in additional trivial or misleading features by adding or removing textures. A study employing cycleGAN used a structural loss to suppress these features, which the authors refer as “imaginary” ability of the GANs especially while encountering new features.[27] Many variants of GAN with diverse characteristics have been proposed for handling the drawbacks of GANs, but several issues persist to be dealt with for future research to take full advantage of the extensive capabilities of GAN-based methods to transform and advance digital pathology.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.