Spectrum of Advancements and Developments in Multidisciplinary Domains for Generative Adversarial Networks (GANs).

The survey paper summarizes the recent applications and developments in the domain of Generative Adversarial Networks (GANs) i.e. a back propagation based neural network architecture for generative modeling. GANs is one of the most highlighted research avenue due to its synthetic data generation capabilities and benefits of representations comprehended irrespective of the application. While several reviews for GANs in the arena of image processing have been conducted by present but none have given attention on the review of GANs over multi-disciplinary domains. Therefore, in this survey, use of GAN in multidisciplinary applications areas and its implementation challenges have been done by conducting a rigorous search for journal/research article related to GAN and in this regard five renowned journal databases i.e. "ACM Digital Library"," Elsevier", "IEEE Explore", "Science Direct", "Springer" and proceedings of best domain specific conference are considered. By employing hybrid research methodology and article inclusion and exclusion criteria, 100 research articles are considered encompassing 23 application domains for the survey. In this paper applications of GAN in various practical domain and their implementation challenges its associated advantages and disadvantages have been discussed. For the first time a survey of this type have been done where GAN with wide range of application and its associated advantages and disadvantages issue have been reviewed. Finally, this article presents several diversified prominent developing trends in the respective research domain which will provide a visionary perspective regarding ongoing GANs related research and eventually help to develop an intuition for problem solving using GANs. © CIMNE, Barcelona, Spain 2021.

Introduction

The inculcation of Machine learning in almost every domain of life being the most popular and effective research domain is a universal reality. It is becoming a dominating and driving domain with novel research avenues with undoubtedly significant numerous applications e.g., computer vision, autonomous driving, speech recognition, image identification, drug discovery, cyber security, anomaly detection etc. The representation of data in machine learning approaches requires appropriate feature extraction and therefore researchers proposed a representation learning [1] approach for feature mining and its onward application for classification. Deep learning [2] is an autonomous feature learning approach by extracting high level abstract features and framing it into very intuitive and meaningful representations. In deep learning [3], numerous tactics have been presented to overcome the challenges and limitations related to hidden layer by employing cascaded hidden layers. There are many well-known deep learning approaches [4] such as Deep Belief Network (DBN) and Boltzmann Machine (DBM) [5], Deep Neural Network (DNN) [6], Auto Encoder with their variants and Convolutional Neural Network (CNN) [5].

Generative adversarial Networks(GANs) have recently accumulated substantial importance in the arena of Generative Models by assuring the ability to produce novel content such that one generative model can generate only data while others can provide an estimate of density function. However, there are some generative models which can generate both data and density function. Generative models unlike discriminative models, successfully aim to understand the underpinning data distribution by learning the fundamental parameters which enables model data analysis, extraction of novel intuitions and synthetic data generation such that conversion of a low dimension input to a high dimension output. The recent emergence of GANs as most dominating category of generative models is based on the aggregate or collective optimization of two neural networks with mutually exclusive objectives. The first network i.e. generator strives to generate synthetic but perceptually convincing data points from some random uniform distribution akin to input representation from a low dimension source space to a high dimension target domain. The second network i.e. discriminator strives to distinguish between the original and fake sample. The behavioral optimization of both the generator and discriminator is based on the discriminators’ output such that weight adjustment for the generator will be required if the discriminator can easily differentiate the fake and real samples. The training process for generator and discriminator is conducted separately such that initially real samples are submitted to discriminator with correct labels, updating the discriminator by adjusting its stochastic gradient while keeping the generator fixed. Subsequently, the generator is updated by adjusting its stochastic gradient while keeping discriminator fixed and the process is repeated until Nash equilibrium is achieved. The model optimization is performed wr.t. a combined loss function for generator as well as discriminator which is given in Eq. 1.

Increasing use of GANs in image processing and image creation [7] can be employed in fashion industry to create imaginary fashion models which obviates the need to hire real fashion model, photographer and makeup artists resulting in significant economy of cost. GANS have also been used in simulating images [8] of astronomical bodies and in 2018 they were used for successful modeling of dark matter distribution in space. They have also been used to visualize effects of climate change [9], face recognition over the years, cartography as well as in fields as diverse as Image Processing to Machine learning and natural language processing (Table 1).

Table 1

Comparison of survey works for application of gans in multidisciplinary domains

Paper author	Year	Areas covered	Focus applications	Multidisciplinary domains	Remarks
Wu [10]	2017	1	Image processing and translation	Not covered	The research paper presents a survey of GANs application in image fusion and associated manipulations
Wang [11]	2017	3	Computer vision, natural language processing and malware analysis	Very limited coverage	The paper covers GANs’ background, fundamental concepts, implementation approaches followed by applications
Zhang [12]	2017	1	Image synthesis	Not covered	The article presents an overview about adversarial training framework and survey of related work
Saifuddin Hitawala [13]	2018	1	Image processing	Not covered	The article study image processing model as well as its variation and their comparative analysis
Qiantong Xu [14]	2018	2	Image processing	Not covered	It presents numerous several representative evaluation parameters and challenges related to the evaluation of those parameters
Zhaoqing. Pan [15]	2019	2	Natural language processing and computer vision	Not covered	Discussed basic theory including the differences among different generative models as well as derived models
Zhengwei Wang[16]	2020	1	Computer vision	Not covered	Conducted a review of GAN-variants and challenges in computer vision
Jie Gui [17]	2020	4	Computer vision, medical and data science	Limited coverage	Covered motivations, mathematical representations and few applications
Proposed work	2020	23	Multidisciplinary and cross domain	Covered	We made an extensive survey of, 5 different research databases and discussed application of GAN over a portfolio of 22 diversified and multidisciplinary domains

While going through the extensive literature review regarding the GANs it was revealed that several reviews for GANs in the area of image vision and processing have been conducted by present but none have given attention on the review of GANs over multi-disciplinary domains Therefore, in this survey, use of GAN in multidisciplinary applications areas and its implementation challenges have been done by conducting a rigorous search for journal/research article related to GAN and in this regard five renowned journal databases i.e. “ACM Digital Library”,” Elsevier”, “IEEE Explore”, “Science Direct”, “Springer” and proceedings of best domain specific conference are considered. By employing hybrid research methodology and article inclusion and exclusion criteria, 62 research articles are considered encompassing 23 application domains for the survey. The major benefactions are: The paper organization is as follows: Sect. 2 narrates methodology including article search strategy and its selection criteria. Section 3 narrates the application of GAN in diverse portfolio of practical domains. Section 4 describe the review analysis along with the problems of GAN. Section 5 narrates the concluding remarks followed by the future direction.

This article presents several diversified prominent developing trends in the respective research domain which will provide a visionary perspective regarding ongoing GANs related research and eventually help to develop an intuition for problem solving using GANs.

This paper provides a unique view of GANs in impressive cross and intersection domains. This research work is also an attempt to provide a unique survey of its kind which has never been studied before in the best of our knowledge and it is not only restricted to domain of natural language and image processing.

Methodology

There are three fundamental approaches to conduct research i.e. qualitative, quantitative and mixed keeping in view problem statement, domain expertise of the researcher, nature of data being analyzed and reporting audience. Qualitative research covers descriptive and brief approach to yield significance and interpretation of the experimental phenomena and have further classification including narrative, ethnography, case study and grounded theory. However, quantitative research employs statistics with further classification including descriptive, cross-sectional, correlational and experimental approaches. Moreover, mixed methodology combines quantitative and qualitative approaches to attain in-depth understanding of the phenomena being probed with further classification including concurrent triangulation, sequential exploratory design, and concurrent nested. The most frequently used research approach is mixed for generative modeling and its application for better output and evaluation. GAN has emerged to be a revolutionary approach for generative modeling in recent years which is evident by the significant number publications covering scope of its advancements in domain of image processing and computer vision. Despite the existing studies, limited research is done regarding its application in other domains and therefore following questions are addressed in this research work: In this research work, the following strategy is used for article searching and selection (Fig. 1).

Fig. 1

Search and selection methodology

Which domains can be the candidate for the application of GAN?

What approaches and tools are being used in the development of GAN based learning?

What are the advantages and limitations of discussed novel GAN based approaches?

The critical step is the manuscript selection for appropriate review and therefore to keep the review process methodologically strong and uniform, a criterion is defined for article selection. Different journals relevant to large number of domains are selected from specific libraries including five renowned journal databases i.e. “ACM Digital Library”,” Elsevier”, “IEEE Explore”, “Science Direct”, “Springer” and proceedings of subject specific top conferences. The collection of aforementioned sources is performed on respective impact factor value and relative ranking. Moreover, keywords used for paper searching comprised of “Applications of GANs”, “Review of GANs applications”, “Benefits of Generative Modeling”, “Advancements of GANs in combination to domains keywords and other tags. The criteria for inclusion or excluding the article is summarized in the Table 2.

Table 2

Manuscript inclusion and exclusion criteria

Inclusion	Exclusion
Criteria
The articles publishing language is English	Articles published in any language except English
Studies published in and after 2016	Articles published before 2016
Work focusing GAN based framework in an applied domain	Work focusing in an applied domain but not using GAN
Articles published in open access journals	Articles with restricted access

In this research, 23 practical applications domains were selected and based on inclusion and exclusion criteria, over 100 studies from selected journals were included in this analysis. The applications domains selected for the application of GAN’s are summarized in the Table 3.

Table 3

Application domains

Image processing	Unmanned aerial vehicles (UAV’s)
Speech recognition	Simulation and modeling
Genetic engineering	Market prediction and forecasting
Drug discovery	NLP—natural language processing (NLP)
Health	Architectural designing
Fault prediction	Road Network generation and path planning
Agriculture	Testing and validation
Music	Software designing and development
Weather forecasting	Fake audio, video and image generation
Sports	Text generation
Internet of things (IoT)	Malware detection

The process of data extraction, analysis and synthesis was done with due diligence and in depth study of all the selected articles with appropriate supervisory review controls. The findings of the review are presented in the following section in the context of established questions keeping in view that given papers are from different domain without any standard theme (Fig. 2) .

Fig. 2

Article screening year distribution

The following section explaining the applications of GAN into computer vision, healthcare, IoT, agriculture, Intelligent transportation, software development, natural language processing, entertainment and cyber security have been summarized (Table 4).

Table 4

Overview Index for GAN based applications

Application	Paper	Year	Description
Image processing	Shrivastava et al. [18]	2017	Simulated and unsupervised learning for synthetic image generation Use image as input instead of random vector
	Yang et al. [19]	2017	Conditional GAN for reconstruction of compressed or corrupted sensing magnetic resonance imaging(CS-MRI) from random under sampled data Generate image with better textures and edges
	Cho et al. [20]	2019	Style transfer approach for image transformation Achieved efficiency and image quality
	Dong et al. [21]	2017	Image translation as well as intra-domain image translation
	Zhang et al. [22]	2016	Conditional augmentation technique for generation of low resolution image from text followed by noise removal Achieved image smoothness and improved quality
	Reed et al. [23]	2016	Synthetic image generation from text description Advantage: image generation with high quality and with multiple objects and variable background
Speech recognition	Sriram et al. [24]	2017	Data driven approach that introduced invariance in encoder embedding with no specialized pre-processing Achieved scalability and robustness
	Donahue et al. [25]	2017	Enhancement of contaminated speed by additive and reverberant noise by employing GAN with Log-mel filter banks instead of wavelet Attained performance and robustness
	Pascual et al. [26]	2017	Application of GAN at waveform level
Unmanned aerial vehicles (UAV’s)	Wang et al. [27]	2018	Noise filtering to avoid information loss during remote sending Benefit: image de-noising
	Qiuhong et al. [28]	2019	High volume data redundancy by applying compression
NLP—natural language processing (NLP)	Li et al. [29]	2018	Text regression model for association of text data and social outcome Advantage: data analysis with limited labelling
	Lin et al. [30]	2017	Rank-Gan for data analysis and quality assessment using rank metric
	Qian et al. [31]	2018	Event factuality identification using Ac-GAN by learning syntactic inform and address imbalance among factuality values Advantage: reduced reliance over annotated text
Health	Che et al. [32]	2017	Hergan for synthetic health data generation with limited electronic health record (HER)
	Hwang et al. [33]	2017	Disease prediction using AC-GAN and stacked auto-encoder
	Rezaei et al. [34]	2018	Semantic segmentation and disease classification by selective weighted loss Advantage: address Data imbalance
Fake audio, video and image generation	Choi et al. [35]	2018	Stargan for fake image generation by using deep CNN Achieved high classification accuracy
	Nataraj et al. [36]	2019	Detection of fake images using co-occurrence matrices along with deep learning Achieved good generalization and very high classification accuracy
Agriculture	Suarez et al. [37]	2017	Strength assessment of vegetation against normalized difference vegetation index (NDI) by applying Conditional GAN
	Barth et al. [38]	2017	Cyclegan for gap reduction between synthetic and empirical image data set Advantage: ease of translation of color and textures
Music	Yang et al. [39]	2017	Midinet—generation of musical notes by using CNN GAN Comparison of midinet was also made with Google’s melodyrnn from scratch Advantage: combine existing melodies as well as generate melodies from multiple channels
	Dong et al. [40]	2018	Misegan—generates symbolic music i.e. piano-rolls of five tracks and four bars i.e. Bass, drums, guitar, piano and strings Proposed three models known as call jamming model, composer model and hybrid model for music generation
	Yu et al. [41]	2019	Simultaneous generation of lyrics-conditioned melody and association alignment between syllables of given lyrics by using conditional deep Lstm generator and discriminator Deep generative model for generation of melody and notes of predicted melody
Weather forecasting	Chen et al. [42]	2018	Scenario generation used for weather forecasting, however errors become more pronounced when the typhoons move into deep sea Advantage: generates wind patterns and weather forecasts based on historic data
	Ruttgers et al. [43]	2018	Predict track of typhoons by using satellite image. If information about surface pressure, velocity and sea surface temperature are added the results can become more accurate Advantage: predict the typhoon center as well as the movement of clouds with certain margins for error
Sports	Jiao et al. [44]	2018	Distinguishes correct performed golf swings Achieved accuracy and precision both in identification as well as classification of golf swings
	Deverall et al. [45]	2017	Conditional GAN for designing athletic shoes based on google gnet Achieved shoes categorization according to their physical attributes as well as functional type
Internet of things (IoT)	Wang et al. [46]	2018	Use of Bayesian methods for Radio Frequency (RF) sensing for IoT Advantage: overcome limitation of limited data availability by introducing an offline stage
	Zhao et al. [47]	2018	Individual identity authentication by applying open-categorical classification model based on gan (occ-gan) Advantage: better results are achieved than other methods like one-class support vector machine (oc-svm) and one-versus-rest support vector machine (ovr-svm)
Genetic engineering	Dizaji et al. [48]	2018	Gene expression profiling by using semi-supervised GAN for expression inference Use landmark genes instead of whole gene expressions
Simulation and modeling	Hassouni et al. [49]	2018	Generating realistic simulation environments that simulates daily activities of users Advantage: generate realistic sensory data that related to daily activities of users
	Pöpperl et al. [50]	2019	Synthetic ultrasonic signal simulation using conditional gans (cgans) Advantage: real like data augmentation for automotive ultrasonic and also adaptive to external influences
Market prediction and forecasting	Tian et al. [51]	2019	A technique for predicting the consumption of energy Advantage: outperforms the standard approaches i.e. information diffusion technology (idt), the heuristic mega-trend-diffusion (hmtd) technology and the bootstrap technique Advantage: scalable to perform forecast for demand of electricity and the traffic supply
	Luo et al. [52]	2018	A technique for predicting the prices of the crude oil using adaptive scales continuous wavelet transform (as-cwt) Advantage: more accurate forecasts as compare to naive forecast (nf) model and other nonlinear models i.e. deep belief networks (dbns)
Drug discovery	Zhavoronkov et al. [53]	2019	Drug discovery using generative modelling. i.e. generative tensorial reinforcement learning Useful for the discovery of new micro molecule kinase inhibitors and DNA damage response (DDR1) inhibitors
Architectural designing	Zheng et al. [54]	2018	Floor plan image identification and creation Floor design images get translated into programmatic patches of colors
	Wang et al. [55]	2019	Double P-buried layers MISFET (DP-MISFET) is proposed Simulated and characteristics are analysed by the Sentaurus TCAD tool
Road network generation and path planning	Albert et al. [56]	2018	Novel technique to simulate real like urban designs fine-tuned with urban land-use inventory Advantage: synthetic urban pattern is formulated to qualitatively regenerate the spatial structures perceived in urban designs
	Mohammadi et al. [57]	2018	Precise and reliable paths for navigation software including wayfinding for disabled people, route identification for evacuations, robotic navigations for autonomous vehicles Advantage: high accuracy of the classification task with high quality of the generated paths is achieved
Testing and validation	Zhang et al. [58]	2018	Unsupervised model for automatic verification and validation of the consistent behavior of autonomous vehicle driving systems Real time validation is also achieved
	Segura et al. [59]	2016	Metamorphic verification and validation approach for identifying unusual behaviors of autonomous vehicle systems along with input validation
	Zhihui Li et al. [60]	2019	Create fuzzing data using Wasserstein GANs (wgans) Advantage: does not require specification of input data Significant for testing of industrial control systems (icss)
Software designing and development	Li et al. [61]	2019	Layoutgan—Wireframe designing i.e. layouts generation of relational graphic elements to wireframe images by modelling geometric relations of different types of two dimensional elements Advantage: introduction of wireframe rendering layer which produce a set of relational graphic controls
	Liu et al. [62]	2018	Treegan for source code generation Advantage: syntax-aware sequence generation
Fault prediction	Gao et al. [63]	2019	ASM1D-GAN a model to identify the faults by extracting features related to faults from real fault samples and create the similar one Advantage: integration of data creation and fault determination
	Zhou et al. [64]	2019	Synthesize vibrational fault samples using a technique of global optimization Advantage: feature extraction of feature using limited number of samples and its effective representation using auto-encoder Filter the non-compliant synthetic samples which are not useful for reliable fault diagnosis
	Zheng et al. [65]	2019	Gan-fp utilizes multiple GANs to create training samples and an inference network in parallel to predict failures for newly crafted samples Improved performance as well as significant socio-economic impact
Text generation	Subramanian et al. [66]	2018	Ability to create sentence outlines using an adversarial model which learns the distribution of sentences in a hidden space persuaded by sentence encoder Advantage: produce real like samples with multinomial sampling
	Liang et al. [67]	2017	Create useful distractors Advantage: achieves comparable performance to a frequently used word2vec-based method for the Wiki dataset
Malware detection	Dahl et al. [68]	2013	Employ random projections to decrease the dimension of the original latent space Achieved improved classification results
	Grosse et al. [69]	2016	To craft real offensive adversarial attacks Introduced additional constraints in the adversarial sample crafting (i) continuous, differentiable input domains are replaced by discrete, often binary inputs; and (ii) the loose condition of leaving visual appearance unchanged is replaced by requiring equivalent functional behavior
	Arjovsky et al. [70]	2017	IDSGAN used to generate malware attacks which can bypass the different intrusion detection systems (IDS) Achieved high degree of evasion against IDS
	Heusel et al. [71]	2017	Framework to target portable executable (PE) anti malware systems in an offensive way Advantage: proved to be an effective model to identify the vulnerabilities of the anti-malware systems
	Arjovsky et al. [72]	2017	Model to generate malware instances for Black-Box Attacks Based on GAN
	Gulrajani et al. [73]	2019	Adversarial sample generation to launch attack against malware classifiers
	Singh et al. [74]	2019	Generative model for malware images that could be used to boost classifier’s performance by performing data augmentation Advantage: leveraged to generate malware images which would alleviate the problem of public sharing of the dataset
	Odena et al. [75]	2016	Class-conditional image synthesis model to segregate generated samples to their respective malware category without any manual intervention
	Anderson et al. [76]	2016	Model to bypass a detector of web domain generation algorithm
	Rigaki et al. [77]	2018	To adapt malware communication to force misclassification of new generation Intrusion Prevention Systems (IPS) Advantage: effective at modifying malware traffic in order to remain undetectable
	Labaca et al. [78]	2019	GAN to inject automatic byte-level perturbations into PE files to fool the classifier
	Kawai et al. [79]	2020	Bypass malware defenders by adding benign to the original malicious code Advantage: resolve the problem of creating an huge collection of APIs to bypass the detectors
BlockChain	Zheng [80]	2020	GANs based technology for exchange of secret key which also overcome the block chain problems of security, recovery of lost key and communication inefficiency Advantage: a new avenue is opened the exchange of secret key which us reliable and adaptive as well as efficient

Image Processing

In [18], authors proposed a model which uses simulated and unsupervised learning instead of using random vectors and synthetic images as inputs. The major challenge faced in image processing is the use of synthetic images in the learning or training phase as it doesn’t produce reliable results when applied over real images. This deficiency is the result of characteristics peculiar to synthetic data as the network learns details that are present only in synthetic images and there is substantial difference between generalizations of actual and synthetic images and therefore results in production of images which are highly realistic. These images are later on evaluated using both qualitative measures as well as through user study. The result of both of these evaluations show a significant improvement over synthetic images without the use of any labeled real data. In [19] the authors presented a strategy for reconstruction of compressed sensing magnetic resonance imaging (CS-MRI). Compared to MRI, CS-MRI uses much less raw data to reconstruct MRI images. CS-MRI has an edge as it can rebuild images from randomly collected fewer samples without suffering consequent quality degradation in images due to nonlinear optimization. The major obstacle for CS-MRI is finding an appropriate reconstruction procedure of a noise free image from random under sampled data. In order to achieve this the authors, propose a new conditional GAN (DAGAN) model for the reconstruction of CS-MRI. A comparison is made of this new method with the baseline methods of image reconstruction and results indicate the dominance of the devised approach over the traditional ones as the images produced have better texture and edges.

In [20] the authors use an approach different from normalization methods but close to style transfer approach for image transformation in GANs which resulted in better efficiency and image quality compared to images produced through normalization techniques. One of the major problems in image processing is the image translation process which requires use of labeled data which is time consuming process. If the image translation can be performed on unlabeled data, it can result in significant cost of economy. In order to address this problem in [21] a two-step method is used for image translation. This method has the advantage that it can perform image translation between two different domains without the use of labeled data which results in significant cost of economy. Zhang et al. [22] a new GAN model StackGAN is proposed for text to image generation which is comprised of dual stages whereas in former stage the model creates an image of low resolution from the textual description while later phase performs the refinement of the image followed by removal of various defects and this results in a high resolution image which is also more realistic. StackGAN uses novel conditioning augmentation technique to achieve smoothness and better results. Reed et al. [23] a new GAN model that is capable of producing images from text descriptions is proposed. The experiments conducted by authors produce result in which images of animal and plants are produced from in depth text deails. The capability of the model to create image with multiple objects and variable backgrounds was also demonstrated.

Speech Recognition

In [24] the authors introduce a new speech recognition model which relies on a data driven approach instead of relying on domain expertise or signal processing assumptions. The framework is scalable and increases robustness as it uses invariance at the encoder embedding level. Experiments show that the new approach has demonstrated good speech recognition capabilities of plain sequential models without requiring particular preprocessing. In [25] the authors increase the effectiveness of GANs to improve speech containing noise of additive kind as well as reverberant noise. Previously one of the main problems faced in the field of speech recognition was that application of GAN only improved the output of a noise free trained automatic speech recognition (ASR) system however failed to increase performance compared to those achieved by conventional multi-style training (MTR). In order to overcome this deficiency, the authors demonstrate that applying GANs on Log-Mel filterbank spectra results in better performance, less computation requirements, increased robustness and achieved a 7% WER improvement relative to the MTR system. The increased robustness is achieved by increasing resistance to reverberant noise whereas improved WER is achieved by appending the GAN-enhanced features to the noisy inputs and retraining. Pascual et al. [26] a new GAN for speech enhancement is proposed which operates at waveform level and the model has the ability to include 28 speakers which was tested against diverse noise conditions. The results demonstrated that the new model was capable of handling the dataset in an efficient manner.

Unmanned Aerial Vehicles (UAV’s)

UAVs or drones are used for a variety of purposes which include military, agriculture, recreation, science, commerce and in capturing data for remote sensing images. Wang et al. [27] the authors try to solve the problem of noise in UAVs. Noise causes loss of content during remote sensing of images by UAVs. The authors discovered the association between noisy and benign images and establish a theoretical model that helps in getting better images by removing noise. The resultant images have clearer and well defined texture and edges. The deniosed images were compared with other images and the results clearly indicated that this method has achieved better results. Another problem faced in image processing by UAVs is the presence of large amount of unnecessary data. The occurrence of hefty volume of data results in problems during post processing phase. In [28] the authors try to overcome this issue through the use of compression techniques to compress the image taken by UAV. This compression is done by an encoder which is placed in front of the generator whereas to improve the quality of compressed image, discriminator is used. This is achieved by training the compressed image along with the real image. The experiments conducted in the paper show in comparison with traditional compression methods this method has achieved better compression results and is also able to achieve hierarchical synthesis of non-essential parts of the image (trees and rivers) with essential components like houses and roads.

NLP—Natural Language Processing (NLP)

In [29] the authors propose a text regression model which associates text data and social outcomes. The advantage of this model is its ability to work with datasets which have limited labeling. In [30] a new GAN RankGAN is proposed which uses policy gradient technique. RankGAN has the ability to analyze and rank human and machine written sentences. RankGAN is thus able to analyze a set of data and indicate their quality through a ranking system. The semantic task of identifying event factuality is of paramount significance in NLP and traditional research deeply depends on annotated texts. The research work [31] proposes a model for mining indispensable features associated with event factuality from raw texts, used those features as input, and subsequently recognizing event factuality by employing GANS with Auxiliary Classification (AC-GAN) which helps to acquire supplementary syntactic content and overcome the disparity among factuality figures.

Health

Recently GAN generated molecules were used to attack protein targets in critical circumstances like cancer, fibrosis and inflammation. GAN generated molecules have also been introduced into mice recently to help in health related experiments. Most of the GAN models used in health sector rely on huge volume of labeled data. In order to overcome this reliance on labeled data in [32] a new model ehrGAN is proposed which can work with limited electronic health records (HER). EhrGAN mimicks real patient records and is able to data and provide plausible labeled data sets. Hwang et al. [33] GANs are used for disease prediction. A combination of auxiliary classifier (AC-GAN) and stacked autoencoder is used for disease prediction with 98.5% accuracy results. Finally, Rezaei et al. [34] the problem of data imbalance is tackled with the help of a new model. Most of the models have a tendency to be biased towards healthy data which needs to be avoided in clinical applications. In the new model this data imbalance is tackled through selective weighted loss and the use of semantic segmentation and disease classification.

Fake Audio, Video and Image Generation

Choi et al. [35] a new scalable approach StarGAN for image translation is proposed and the core benefit of StarGAN as compared to other models is that can be used for multiple domains whereas previously a new model had to be built for every image pair belonging to different domains. Experiments conducted show the flexibility as well as better quality of images produced by StarGAN. A major problem faced in Image generation by GANS is the creation of fake images. Nataraj et al. [36] a new method has been devised for detecting fake images by devising a deep CNN based model. Experiments conducted on a dataset of more than 56,000 images show that this model was able to achieve 99% accuracy in classification of images.

Agriculture

GAN’s have also been applied in agriculture. In [37] the authors have used a conditional GAN architectural model to calculate normalized Difference Vegetation Index (NDVI) which is helpful to determine the health and quality of vegetation. Vegetation Indexes are dependent upon land cover, vegetation, surface elevation, temperature, soil reflectance and vegetation density. At present over forty vegetation indexes are in use. NDVI is useful in predicting the yield of crops based on their present condition and biomass. The authors have used a single spectral band and evaluated three different schemes (Triplet GAN, Flat and Siamese). A Near Infrared (NIR) image of crops was taken which was later used to calculate NDVI. Experiments showed good success in acquiring a precise estimate of NDVI however, one drawback was that a comparison with results obtained from previously used techniques couldn’t be made due to the novelty of this approach. In the field of agriculture robotics there is a requirement for large synthetic datasets. Manual modeling is used to optimize such large synthetic datasets by improving the color, texture and geometry of these 3D modeled plants with a residual gap between synthetic and empirical content. In [38] the authors have used a cycle GAN to bridge the gap between synthetic and empirical image dataset of agricultural crops. This is done by increasing the realism of synthetic images. The authors used a dataset of 10,500 synthetic images, 225 unlabeled empirical images and 50 empirical annotated images. Experiments conducted by authors showed that there was improvement in the optimization of synthetic data in both qualitative and quantitative terms. It was also shown that certain factors like color and texture of synthetic images can be more easily translated as compared to certain other features like plant morphology which can’t be translated.

Music

GANs are also increasingly applied in the field of music. More specifically the have been applied to generate musical notes as Yang et al. [39] where authors suggest a CNN based GAN i.e. Midinet for the generation of musical notes. MIDINET has also the capability of learning (bothe generator and distributor) the distribution of musical notes as an iterative process. The new music generation model can generate melodies from scratch, combine existing melodies and is also capable of generating melodies from multiple channels. Depending upon the input and specifications this model is capable of generating different types of music. A comparison of MIDINET was also made with Google’s MelodyRNN model and it was found that MIDINET gives comparable performance. Dong et al. [40] another musical model (MiseGAN) that generates symbolic music is proposed. The authors propose three models which they call Jamming model, composer model and hybrid model. The dataset used for the training of these models consist of one hundred thousand bars of rock music. The model is capable of generating piano-rolls of five tracks and four bars i.e. bass, drums, guitar, piano and strings. However as admitted by authors themselves the proposed model needs a lot of improvement as musically and aesthetically it is still far behind music generate by humans. Yu and Canales [41] a novel model for melody generation from lyrics is proposed, comprising of a conditional deep LSTM generator and discriminator subject to lyrics. Furthermore, simultaneous generation of lyrics-conditioned melody and association alignment between syllables of given lyrics and notes of predicted melody is also achieved. The proposed lyrics-to-melody generative model made it possible to infer plausible and tuneful sequences from lyrics.

Weather Forecasting

GAN’s are now also increasingly used in the field of weather forecasting. A recent example of this is [42] in which a new data driven (consisting of wind and solar time data sets) method is proposed for scenario generation used for weather forecasting. This method generates wind patterns and weather forecasts based on different sets of input consisting of labeled data. Both event based (windy or rainy day) and time based (solar generation on a particular day of the year) event sets can be used as inputs. GAN was able to generate scenarios based on historic data sets of weather collected from multiple sites. The results of the paper clearly demonstrate that GANS are very suitable for scenario generation of wind and solar. A further use of GAN was made Rüttgers et al. [43] where authors propose a model which can predict track of typhoons by using satellite images. These satellite images taken at 6 h interval and submitted as inputs for GAN. The model was able to predict the typhoon center as well as the movement of clouds with certain margins for error. In case of typhoon center prediction, the errors observed were less than 80 km in 42:4% cases, within a range of 80–120 km for 32:1% cases and above 120 km for 25.5% cases. These errors become more pronounced when the typhoons move into deep sea, are about to hit mainland or sudden change of their course. If information about surface pressure, velocity and sea surface temperature are added the results can become more accurate.

Sports

Jiao et al. [44] a new GAN model is proposed which takes as data input swings performed by different golf players and distinguishes correct performed golf swings from incorrectly performed golf swings. The input data consists of 13 signal and each signal in turn comprises of 1500 data samples. The model tells in the output the player to which the golf swing belongs and whether the swing belongs to correct or incorrect category. The evaluation results prove that the proposed approach has achieved accuracy and precision in identification of golf swings as well as in its classification of them in the category of correct or incorrect swings. Deverall [45] a conditional GAN is used to design athletic shoes. A dataset consisting of 50,025 images of shoes is used. In the dataset shoes are further categorized according to their physical attributes as well as functional type. In the first step a GAN is designed and then trained on several athletic shoes. In the second step, a classifier is built based on GoogLegNet. This classifier helps to classify the shoes according to their functional types. In the third step a binary classifier for each value of dataset was created. The biggest hurdle faced in this entire process was the vague and subjective nature of attribute comparison dataset. The authors emphasize the need for an effective attribute classifier as future work which can help to enhance the functionality of this model.

Internet of Things (IoT)

Wang et al. [46] a GAN is proposed for RF sensing for IoT and proposed model was evaluated for fingerprinting, event recognition and sign monitoring. The framework is a two phased model where the first is offline stage in which training data is used to train GAN. When the training has been completed then in the second stage this trained data is fed as input to GAN which uses this data for prediction by using different methods such as Bayesian methods. In [47] authors construct an open-categorical classification model based on GAN (OCC-GAN) to solve the problem faced by traditional classifiers in individual identity authentication. This model achieves better recognition results due to auxiliary training and also increases the efficiency. Through experiments it is proven that OCC-GAN achieves 90 percent recognition results with a recall rate of 97 percent. These results are better than other methods like one-class support vector machine (OC-SVM) and one-versus-rest support vector machine (OvR-SVM) but major limitation is small volume of training data. Moreover, to improve the training there is a need to consider other methods like Auxiliary Classifier GAN (ACGAN) for training purposes.

Genetic Engineering

GAN has also been used for gene expression profiling and the biggest challenge faced in the field gene profiling is cost associated with genome profiling of very large libraries. A remedy for aforementioned problem is to use landmark genes instead of complete gene expressions due to its complexity and the cost of process. Dizaji et al. [48] the authors propose a new semi-supervised GAN for expression inference model and therefore instead of taking a completely supervised approach which is dependent upon a huge volume labeled dataset consisting of paired landmark and target gene expressions, inexpensive unlabeled content i.e. landmark genes are used. The GAN synthetic data is used as additional training pairs to bring enhancement in the training performance and the use of semi supervised learning also results in more reliable and better training of inference data.

Simulation and Modeling

GAN has also been used for simulation and modeling and it is known fact for reinforcement learning (RL) algorithms, simulated environments are very worthwhile testbeds but they essentially require to have an appropriate level of realism. The simulator shall have access to real content to generate real while appropriately addressing the privacy issues. In [49] the authors propose to use GANs for generating realistic simulation environments and applied a prevailing simulator to simulate routine tasks of users and to create realistic sensory logs that supplements such routine tasks. The simulator can easily be trained and tested by the GAN based approach and assessment demonstrated same level of performance on synthetic data as delivered real dataset. In [50] the authors presented a novel using conditional GANs (cGANs) method for synthetic ultrasonic signal simulation and as per the authors prerogative, it is the foremost data augmentation technique for automotive ultrasonics which is also adaptive to external influences. The performance of cGANs will surely uplift realistic environment simulation to a new horizon.

Market Prediction and Forecasting

GAN has also been used for prediction of consumption and market trends including price. Prediction of energy consumption is very vital for good energy management, power distribution planning and optimized resource utilization. In [51], the authors proposed a parallel systematic prediction scheme for the calibrating consumption of energy which makes use of a limited number of the original content to generate the synthetic data. Moreover, it constitutes a hybrid dataset comprising the original and synthetic data to train the model to be used for prediction. The results established that the proposed technique outperforms the prevailing baseline schemes i.e. information diffusion technology (IDT), the heuristic Mega-trend-diffusion (HMTD) as well as bootstrap method. The proposed organization can also be enhanced to predict the load of electricity load as well as its flow. The paper [52] proposes a novel approach based on a supervised GANS to predict the price of crude oil using Adaptive Scales Continuous Wavelet Transform (AS-CWT). The supervised GAN based model predicted relatively precise and accurate forecasts in comparison to naive forecast (NF) and other nonlinear model i.e. Deep Belief Networks (DBNs) particularly while interacting with low volume of data. In this approach Continuous Wavelet Transform (CWT) is applied to decompose price of oil commodity into various time oriented components, such as the months, years weeks and sequence of days so that the resultant values of time series can be utilized as inputs for prediction model. The forecasting performance is improved by the application of adaptive scales in the CWT method. In [81] authors presented a GAN based chest x-ray detection based model for detection of novel coronavirus COVID-19 associated pneumonia for a small dataset. To validate the usefulness of the model, authors use limited volume of the original dataset and utilized 90% of synthetic images for model training.

Drug Discovery

Drug discovery is an exhaustive and time taking process. It starts with testing thousands of molecules to just explore a few lead-like compounds which are likely to be useful. But, among the explored ones very small percentage of those qualified clinical trials. The advanced utilization of GANs in the domain of drug discovery is an inspirational illustration of cutting edge application of AI in all facets of life specially the pharmaceutical industry. The article [53] presents a substantial advancement drug discovery using Generative modeling. The authors devised a model i.e. generative tensorial reinforcement learning (GENTRL), has enabled the discovery of new minor molecule i.e. kinase inhibitors, DNA damage response (DDR1) inhibitors.

Architectural Designing

The advancements in artificial intelligence has also introduced into the design field, resulting in the improvement of computer-aided design. The authors in [54] proposes generation and recognition of floor plan using GAN such that images of the floor plan processed by GAN based model can be translated into programmatic patches of colors. Wang et al. [55], a performance-improved AlGaN-/GaN-Based metal–insulator-semiconductor field effect transistor (MISFET) with double P-buried layers MISFET (DP-MISFET) is proposed, simulated and attribute analysis is conducted by the Sentaurus TCAD tool.

Road Network Generation and Path Planning

In [56] the author proposes a novel approach to simulate hyperrealistic urban designs using GANS which is trained with a dataset of inventory used by urban land globally. A synthetic urban pattern is formulated that qualitatively duplicates the spatial organization logged in urban designs. A collection of 30, 0000 training samples of cities was used and results demonstrated that a basic, unconditional GAN based model can create real like urban designs that capture the impressive diversity of urban forms. In [57], the author proposed the use of GANs to acclaim correct and reliable tracks for navigation applications including wayfinding for disabled people, path planning for evacuations, robotic navigations as well as navigation for autonomous vehicles. The model equipped users with synthetic paths that assisted navigation from local environment to reach a desired destination and notably impressive classification accuracy is achieved with reliable synthetic paths.

Testing and Validation

In [58] authors proposed a Deep neural network model that works in unsupervised manner for online verification and validation of self-governing vehicle driving system. The model generates various driving scenes along with various weather conditions next to reality using GAN. In [59], the authors applied metamorphic verification and validation approaches for the identification of unusual patterns and behaviors of self-directed vehicle driving systems as well as validation of input images. Fuzzy testing is another important approach for the identification of software vulnerabilities and errors and in such kind of testing a huge volume of input data is essentially required by the target program and therefore, a critical requirement of fuzzing evaluation is the generation of fuzzing samples. Several customary approaches to create fuzzing data samples have devised including model and random fuzzing data generation. These practices essentially demand the detailed description of input data and its corresponding representation or perform reverse engineering to conduct the said task which is also a challenge. In [60], the authors proposed a novel scheme for generation of fuzzing data using Wasserstein GANS which does not require specification of input data and corresponding representation. Thus, the proposed method is significant for industrial control systems (ICSs) testing.

Software Designing and Development

The software graphical design is imperative for graphical interface and is n significant interactive tool for communication between development team and software user analogous to magazine layouts to web design. A wireframe is a 2D visual and graphical illustration regarding the layout of web page with special attention page space utilization addressing the content allocation and prioritization, availability and placement of features as well as overall behaviors. In [61], the authors proposed GANS for generating graphical design layouts of associated interface controls by learning representation of geometric relations of various kind of graphical controls or elements. The model can generate images in pixel-level i.e. can render a set of relational graphic controls unlike customary GANs. Moreover, a rendering layer was devised to rasterize the synthetic visual interface controls to wireframe images, leaving it appropriate to influence CNNs as discriminator for optimized layout. Beside wireframe layout structuring, GAN can impressively be utilized for generation of source code generation to assist the job of software programming and development. In numerous real-life applications, there is a requirement to create sequences in a formal computer language with the limitation of language grammar. For instance, to validate the operational efficiency of a database, a collection of SQL queries may be generated similar real queries but also comply the language syntax of the target database. Creating such sequences is a tedious job and Liu et al. [62], the authors devised a GAN based syntax-aware i.e. TreeGAN for generation of sequences. The experimental results showed that the proposed technique outperforms the state of the art adversarial learning approaches for generation of syntax-compliant sequences.

Fault Prediction

The extraordinary potential of feature representation abilities of deep learning can be very helpful for diagnosis of fault and errors. Gao et al. [63] a novel GAN based approach i.e. ASM1D-GAN for fault diagnosis is presented and is composed of one dimensional CNN, as well as GAN and a classifier for faults. The model has integrated capabilities of fault diagnosis and data synthesis by processing the features extracted by samples of natural faults. The experimental assessment demonstrated the effectiveness of both capabilities i.e. fault diagnosis and data synthesis. Zhou et al. [64] a novel GAN based model is devised by applying method of global optimization to synthesize vibrational fault samples. The generator is based on auto encoder for the feature learning, representation and subsequent generation of fault sample whereas the discriminator is intended to filter the non-complaint synthetic samples i.e. samples which are not very useful for identification of fault. The effectiveness of the proposed approach is well demonstrated. In [65] the author proposed a GAN based method for prediction of failure and the GAN-FP model employs the concept of multiple GAN networks for simultaneous generation of input samples and construction of an inference network for prediction of failures for synthetic samples. The tuning of inference network is performed by optimizing an objective function of weighted loss objective considering only original failure and non-failure samples followed by the tuning of model using a next GAN to ensure the uniformity between real and synthetic samples. This novel approach will not only improve performance of modeling process but also have substantial socio-economic impact.

Text Generation

Latest developments in generative models is determined by two standards i.e. autoregressive and adversarial models. In [66] the authors devised a hybrid approach with the intent of learning generative models of textual processing. The model creates an abstract outline for sentence and afterwards create words in a sequentially manner while depending on outline as well as the previous outputs. The sentence outlines are created using an adversarial model which learns the distribution of sentences in a hidden space persuaded by sentence encoder which ensures robust and informative conditioning for the autoregressive phase. The quantitative assessments showed that attributes extracted from synthetic outlines can drive the autoregressive model to generate real like samples, comparable to stat of the are existing models even with multinomial sampling for high temperatures. Distractor creation is a critical activity for generation of questions like fill-in-the-blank. In [67], the authors propose a GAN based generative model to generate intelligent distractors by utilizing context information and not relying on the correct answer and therefore is entirely different from existing ontology or similarity based method. The training is done over on the corpus of Wikipedia, predict Wiki entities as distractors and is validated on two datasets comprising of biology questions acquired from Wikipedia as well real educational institutional exams. Experimental results showed that context-based approach attained impressive performance in comparison to a commonly used word2vec-based approach for the Wiki dataset.

Malware Detection

GAN’s are also increasingly used in the field of malware detection i.e. to attack protected systems and highlight the various weaknesses and blind spots of the anti-malware defense systems. An adversary can bypass the malware classifier or the defense system by making slight perturbation to malware file that retains its malicious nature but made to be classified as benign [68, 69]. Arjovsky et al. [70] a new framework IDSGAN is proposed which is used to generate malware attacks which can bypass the different intrusion detection systems. IDSGAN consists of a generator and a discriminator where the generator generates malicious traffic and the discriminator simulates the detection system. Experiments show that IDSGAN achieves high degree of evasion from traditional intrusion detection systems. Heusel et al. [71] a framework is proposed for targeting anti malware engines by framing relatively realistic offensive situations and the model plays a number of games against the anti-malware defense mechanism to improve its learning graph. Though the evasion rates were not substantial but the model was successful in pointing out a number of deficiencies of the anti-malware model and was used by Weiwei Hu and Ying Tan in 2017 to propose a model for Black-Box Attacks based on GAN [72]. In [73] authors proposed the use GAN for the generation of adversarial samples to attack against malware classifiers. Singh et al. [74] presented a GAN based scheme for generation of malware images that could be used to boost classifier’s performance by performing data augmentation. Additionally, it can be leveraged to generate malware images which would alleviate the problem of publicly sharing the dataset. Odena et al. [75] proposed mechanism of class-conditional image synthesis model to segregate generated samples to their respective malware category without any manual intervention, where the generator is conditioned with class label and the discriminator is tasked to predict the class label.

In DeepDGA [76] the authors attempt to bypass a detector of web domain generation algorithm that identifies human-generated domains from its automatically generated counterparts. Rigaki et al. [77] proposed to adapt malware communication to force misclassification of advanced systems of Intrusion Prevention. The malicious code was customized to duplicate the network traffic of the chat application and proposed work suggested that GANs can be successful at modifying malware traffic in order to remain undetectable. The authors implemented a GAN to inject automatic byte-level perturbations into PE files [78]. The authors proposed an approach to bypass detectors by inculcating benign features to the malicious code [79]. The approach introduced a loss calculation Layer to Generator to resolve the issue of generating massive number of APIs for detector avoidance.

BlockChain

The authors [80] presented a GANs based key secret-sharing technology to overcome blockchain problems of security, lost key recovery and communication efficiency. The proposed approach deals the secret-sharing process as a classification subject and main theme is to consider visualization of secret as an image for the exchange of secret key. The proposed scheme provides a novel avenue for the sharing of secret key scheme and also demonstrated results show that the apporach is reliable, adaptive and efficient in communication.

Review Analysis

In this research paper, contrary to other research surveys which are restricted to domains of computer vision, image processing and natural language processing, holistic view of GAN applications have been presented to date. A gap was realized that several surveys have been done regarding GANs but none have focused on the review of GANs over multi-disciplinary domains. Therefore, an extensive survey is conducted in this research work and utility of GANs in diversified domains such as healthcare, IoT, agriculture, Intelligent transportation, drug discovery and pandemic control like covid-19 software development, natural language processing, entertainment and cyber security have been summarized.

Problems of GAN

In this section, we highlight the problems of the generative models faced by the researchers.

Training of GANs is a challenge due to establishment of the equilibrium [82], however, methods have been proposed for making it possible but it is still an active research area [83] and a problem

Lack of benchmarking evaluation criteria for assessment of the stability of the GAN [84] and similarly the establishment of equilibrium progress is underway in this regard but it is still a menace [85].

Convergence is difficult to achieve as both discriminator and the generator models updates their cost function independently. Significant research has focused on identifying improved training algorithms [86] for GANs as well as achieving improved conceptual comprehension of training dynamics [87].

The generator flops due to production of limited varieties of samples. It is called mode collapse [88]. A partial is common but a complete collapse is rare. Reference [89-91] integrate samples created by various models to overcome the limitation.

The problem of Diminished gradients [82] exists. It is possible that the discriminator performs in an outstanding manner that generator’s gradient diminished and ends up with no learning or learning is very slow. On the other side, if discriminator does not perform well then the generator does not have precise and reliable input and the loss function cannot present the true picture. The paper reference [92] proposed a model for the problem of diminishing gradient.

Highly sensitive to the hyper parameter selection [93]. Reference [94] addressed the issue.

Overfitting due to in equilibrium [95, 96] unbalance between both discriminator and the generator models. Therefore, detecting [97, 98] and overcoming overfitting is a good problem area.

Lack of a good objective function is another problem of GAN [99]. The lack of god objective functions this makes it very challenging to perform comprehensive evaluation of models.

Conclusion and Future Direction

GAN has tremendous potential in generative modeling and its application in numerous domains and therefore this review focuses on the identification of those applications. Although a lot has been done in this domain, there is still a desperate need for research in the GAN based applications (Fig. 3).

Fig. 3

Problems of GAN

Future Challenges

Keeping in view literature studied, it is found that generative models has the following challenges:

The research [100] in adversarial machine learning, its goal and capability was proposed in [101]. The authors [102] crafted an inversion attack based on the linear regression over a bespoke but vulnerable drug information system. Current research has shown that the counterintuitive attributes of DNNs affects their security and regardless of considering adversarial samples in training models to improve the robustness of model [103], solutions are still vulnerable to overcome the aforementioned risk. Therefore, secure generative model is a dire need of time and research on such avenue will be very promising e.g., Bayes deep networks with prior information [104]. Overdue security introduces overheads and also affect generalization performance of model that subsequently challenges their application. Hence, moderate balance is recommended to facilitate the practical usage.

Currently, formally standardizing [105] assessment techniques for generative models is still at the initial stage. There are no standard metric available as well. Therefore, it is a challenge for establishing a generally accepted and well-articulated standard defining the evaluation procedure of generative models as well evaluation criteria.

GANs are very famous for synthetic data generation, however, models designed for synthetic data generation have notable limitations. The synthesized data can represent multiple attributes of real data, but does not exact clone of the real data. Moreover, any processing over synthetic data requires verification [106] against the counterpart real dataset. Furthermore, there is requirement of a standard verification procedure to conduct test and perform the quantitative comparative analysis of real and synthetic outputs. The proper training of model must be ensured to guarantee that it is not creating the expected outputs because of assumptions made for synthetic data. Due to the large variety, volume and velocity of data, it is not possibility to generate universal synthetic data generation tool however generation of bespoke synthetic datasets can be the achievable target.

Generative models learn patterns in the real data during the data generation but presence of any noise may be invisible to them that can be a critical issue [106].

Robustness of generative model is also a big challenge. The quality of synthesized data heavily relies on the superiority of the model being used for the process as the generative models can be vulnerable to statistical noise i.e. modifications that can fool the model to misclassify the data [107]. Therefore, such models cannot be utilized in production environment if they are not immune to adversarial attacks.

Optimization of generative models [108], generalization performance and overhead is required to design optimized learning algorithms.

Maturity of generative adversarial learning [109, 110] in the context of quantum computing has been theoretically presented and demonstrated to possess the potential of revealing an unsual benefit over its conventional counterpart can be a future challenge.