On Catastrophic Forgetting and Mode Collapse in Generative Adversarial Networks
CCatastrophic forgetting and mode collapse in GANs st Hoang Thanh-Tung
Applied Artificial Intelligence InstituteDeakin University [email protected] nd Truyen Tran
Applied Artificial Intelligence InstituteDeakin University [email protected]
Abstract —In this paper, we show that Generative AdversarialNetworks (GANs) suffer from catastrophic forgetting even whenthey are trained to approximate a single target distribution.We show that GAN training is a continual learning problemin which the sequence of changing model distributions is thesequence of tasks to the discriminator. The level of mismatchbetween tasks in the sequence determines the level of forgetting.Catastrophic forgetting is interrelated to mode collapse and canmake the training of GANs non-convergent. We investigate thelandscape of the discriminator’s output in different variantsof GANs and find that when a GAN converges to a goodequilibrium, real training datapoints are wide local maxima of thediscriminator. We empirically show the relationship between thesharpness of local maxima and mode collapse and generalizationin GANs. We show how catastrophic forgetting prevents thediscriminator from making real datapoints local maxima, andthus causes non-convergence. Finally, we study methods forpreventing catastrophic forgetting in GANs.
Index Terms —GANs, generative, catastrophic forgetting, modecollapse
I. I
NTRODUCTION
GANs [1, 2] are a powerful tool for modeling complexdistributions. Training a GAN to approximate a single targetdistribution is often considered as a single task. In this paper,we introduce a novel view of GAN training as a continuallearning problem in which the sequence of changing modeldistributions are considered as the sequence of tasks. Wediscover a surprising result that GANs suffer from catastrophicforgetting, a problem often observed in continual learningsettings [3]. Catastrophic forgetting (CF) in artificial neuralnetworks [4, 5, 6] is the problem where the knowledgeof previously learned tasks is abruptly destroyed by thelearning of the current task. When a GAN suffers from CF,it exhibits undesired behaviors such as mode collapse andnon-convergence.In section III, we show that GAN training is actually acontinual learning problem and demonstrate the CF problemon a number of datasets. We show that catastrophic forgettingand mode collapse [1] are two different but interrelatedproblems and together, they can make the training of GANsnon-convergent (section III-B, IV-B). To avoid mode collapseand improve convergence, it is important to address the CFproblem. We identify 2 factors that causes CF in GANs: 1)Information from previous tasks is not used in the current task,2) Knowledge from previous tasks is not usable for the currenttask and vice versa. Our findings shed light on how to avoid catastrophic forgetting to learn the target distribution properly(Section V).In section IV, we investigate the effect of CF and modecollapse on the landscape of the discriminator’s output. We findthat when a GAN converge to a good local equilibrium withoutmode collapse, real datapoints are wide local maxima of thediscriminator. We show that the sharper the local maxima are,the more severe mode collapse is. Section IV-B shows thatwhen CF happen, the discriminator is directionally monotonic.A GAN with a directionally monotonic discriminator does notconverge to an equilibrium. The fact confirms that CF is acause of non-convergence.Section V explains how state-of-the-art methods for stabi-lizing GANs such as Wasserstein GAN [7, 8], zero-centeredgradient penalty on training examples (GAN-R1) [9], zero-centered gradient penalty on interpolated samples (GAN-0GP)[10], and optimizers with momentum, can prevent CF andmode collapse. Finally, we introduce a new loss function thathelps preventing CF while adding zero computational overhead.
Contributions:
1) We detect the CF problem in GANs.2) We show the relationship between CF, mode collapse,and non-convergence.3) We study the relationship between the sharpness of localmaxima and mode collapse.4) We show that CF tends to make the discriminatordirectionally monotonic around real datapoints.5) We identify the causes of CF and explain the effectivenessof methods for preventing CF in GANs.II. R
ELATED WORKS
Convergence.
Prior works on the convergence of GANsusually consider the convergence in parameter space [9, 11,12, 13]. However, convergence in parameter space tells littleabout the quality of the equilibrium that a GAN converge to.For example, Thanh-Tung et al. demonstrated that TTUR [12]can make GAN converge to collapsed equilibrium. ConsensusOptimization [13] can introduce spurious local equilibria withunknown properties to the game.We directly study the behaviors of GANs in the data space.By analyzing the discriminator’s output landscape, we find thatwhen a GAN converges, real datapoints are local maxima ofthe discriminator. We discover the relationship between thesharpness of local maxima and mode collapse, generalization. a r X i v : . [ c s . L G ] M a r atastrophic forgetting. Seff et al. [14] studied the standardcontinual learning setting in which a GAN is trained to generatesamples from a set of distributions introduced sequentially. Theproblem is solved by the direct application of continual learningalgorithms such as Elastic Weight Consolidation (EWC) [3]to GANs. Liang et al. [15] independently came up with asimilar intuition that GAN training is a continual learningproblem. The paper, however, did not study the causes andeffects of the problem and focused on applying continuallearning algorithms to address catastrophic forgetting in GANs.We focus on explaining the causes and effect of the problemand its relationship to mode collapse and non-convergence.III. C
ATASTROPHIC FORGETTING PROBLEM IN
GAN S A. GANs training as continual learning problems
Let us consider a GAN with generator G ( · ; θ ) : R d z → R d , acontinuous function with parameter θ ∈ R m ; and discriminator D ( · ; ψ ) : R d → R , a continuous function with parameter ψ ∈ R n . G transforms a d z -dimensional noise distribution p z to a d -dimensional model distribution p g that approximates a d -dimensional target distribution p r . D maps d -dimensionalinputs to -dimensional outputs. Let L D be the loss functionfor D , L G be the loss function for G (Table I). In practice, G and D are neural networks trained by alternating SGD [1].At each iteration of the training process, G is updated tobetter fool D . p tg , the model distribution at iteration t , isdifferent from the model distribution at the previous iteration p t − g and the next iteration p t +1 g . The knowledge required toseparate p tg from p r is different from that for the pair { p t − g , p r } . { p t − g , p r } and { p tg , p r } are two different classification tasksto the discriminator. The sequence of changing model distri-butions (cid:8) p ig (cid:9) Ti =1 and the target distribution p r form a sequenceof tasks (cid:8) T i = { p ig , p r } (cid:9) Ti =1 to the discriminator. Becausethe generator at iteration t , G t , can only generate samplesfrom p tg , D t , the discriminator at iteration t , cannot accesssamples from previous model distributions p We beginby analyzing the problem on the 8 Gaussian dataset, a datasetgenerated by a mixture of 8 Gaussians placed on a circle. InFig. 1, red datapoints are generated samples, blue datapointsare real samples. The discriminator and generator are 2 hiddenlayer MLP with 64 hidden neurons. ReLU activation functionwas used. p z is a 2-dimensional standard normal distribution.SGD with constant learning rate of α = 3 × − was usedfor both networks. The vector at a datapoint x shows thenegative gradient − ∂ L G / ∂ x . The vector shows the directionin which L G decreases the fastest. The length of the vectorcorresponds to the speed of change in L G . Because the gradientfield is conservative, the the difference between the loss of twodatapoints x and x is: L G ( x ) − L G ( x ) = (cid:90) C v · d s (1)where v = − ∂ L G / ∂ x and C is a path from x to x . For thevariants in Table I, ∂ L G / ∂ x only depends on x and D . Becausedecreasing L G in these GANs corresponds to increasing D ( x ) ,going in the direction of − ∂ L G / ∂ x increases the score D ( x ) .Let y = G ( z ) , z ∼ p z be a fake datapoint. Updating y with SGD with a small enough learning rate will move y in the direction of − ∂ L G / ∂ y by a distance proportional to (cid:107)− ∂ L G / ∂ y (cid:107) . If the discriminator is fixed, then SGD updateswill move y along its integral curve, in the direction ofincreasing D ( y ) . Fig. 1a - 1d show the evolution of a GAN-NS on 8 Gaussiandataset. In Fig. 1a - 1c, the discriminator assigns higher scoreto datapoints that are further away from the fake datapoints,regardless of the true labels of these points. This is shown bythe gradient vectors pointing away from the fake datapoints.The integral curves do not converge to any real datapoints.If D is fixed, updating G with gradient descent makes p g diverges. Because gradients w.r.t. different fake datapoints have In practice, gradient updates are not applied to y but to the generator’sparameters. Because the generator also minimizes L G , gradient updates to thegenerator move y in a direction that approximates − ∂ L G / ∂ y . − ∂ L G / ∂ y isa good approximation of the direction that y will move in the next iteration. D L G WGANGP − E x ∼ p r [ D ( x )] + E z ∼ p z [ D ( G ( z ))] + λ E u [( (cid:107) ( ∇ D ) u (cid:107) − ] − E z ∼ p z [ D ( G ( z ))] where u = α x + (1 − α ) y ; x ∼ p x , y ∼ p g , α ∼ U (0 , GAN-NS E x ∼ p r [ − log( D ( x ))] + E z ∼ p z [ − log(1 − D ( G ( z )))] E z ∼ p z [ − log( D ( G ( z )))] GAN-R1 E x ∼ p r [ − log( D ( x ))] + E z ∼ p z [ − log(1 − D ( G ( z )))] + λ E x ∼ p r [ (cid:107) ( ∇ D ) x (cid:107) ] E z ∼ p z [ − log( D ( G ( z )))] GAN-0GP E x ∼ p r [ − log( D ( x ))] + E z ∼ p z [ − log(1 − D ( G ( z )))] + λ E u [ (cid:107) ( ∇ D ) u (cid:107) ] E z ∼ p z [ − log( D ( G ( z )))] where u = α x + (1 − α ) y ; x ∼ p x , y ∼ p g , α ∼ U (0 , TABLE I: Loss functions of GAN variants considered in this paper. (a) Iteration 3000 (b) Iteration 3500 (c) Iteration 3600 (d) Iteration 20000 (e) Iteration 1000 (f) Iteration 2500 (g) Iteration 5000 (h) Adam. Iteration 1500 Fig. 1: (a) - (d) catastrophic forgetting in GAN-NS trained on the 8 Gaussian dataset. (e) - (g) GAN-R1 with λ = 10 .GAN-0GP and WGAN-GP exhibit similar behaviors on this dataset. (h) GAN-NS trained with Adam. Viewing on computer isrecommended.the same direction, almost all of fake datapoints move in thesame direction and do not spread out over the space. Becauseof CF, the generator is unable to break out of mode collapse. Inside the green box (Fig. 1a), gradients at all datapointshave approximately the same direction. The loss L G decreases(the score D ( · ) increases) monotonically along the directionof the green vector u , a random vector that points away fromthe fake datapoints. We have the following observation: Observation 1. In a large neighborhood around a realdatapoint, L G (and therefore, D ( · ) ) is directionally monotonic. A theoretical explanation to this phenomenon is given in Sec.IV-B. Because fake samples in Fig. 1a-1d are concentrated in Graphically, we see that the angles between the green vector u and v = − ∂ L D / ∂ x are less than ° for all x in the box. Thus, the dot product v · d u is positive. The line integral in Eqn. 1 is positive for x , x in thebox that satisfy x = x + k u , k > . L G monotonically decreases alongthe direction of u . We say that L G is monotonic in direction u . a small region (i.e. mode collapse), D can easily separate themfrom distant real samples and does not learn useful featuresof the real data. We say that D catastrophically forgets realsamples that are far away from the current fake samples. Modecollapse and CF are interrelated, one problem makes the othermore severe. In Fig. 1b, fake datapoints on the right of the red box havehigher scores than real datapoints on the left, although in Fig.1a, these real datapoints have higher scores than these fakedatapoints. Going from Fig. 1a to 1d, we observe that thevectors’ directions change as soon as fake datapoints move.The phenomenon suggests that information about previousmodel distributions is not preserved in the discriminator . As D t tries to separate p tg from p r , it assigns low scores to regionswith fake samples and higher scores to other regions. Because D t does not ’remember’ p We performedexperiments on real world datasets to confirm the existenceof CF in GANs. We visualize the landscape around a realdatapoint x by plotting the output of the discriminator alonga random line through x . We choose a random unit vector ˆ u ∈ R d , (cid:107) ˆ u (cid:107) = 1 and plot the value of the function f ( k ) = D ( x + k ˆ u ) (2)for k ∈ [ − , . We use the same ˆ u for all images inFig. 2. We choose to visualize D ( · ) instead of L G because L G explodes if D ( · ) (cid:28) . The quality of the image x + k ˆ u decreases as | k | increases. A good discriminator D ∗ shouldassign lower scores to samples with lower quality. D ∗ ( x ) should be higher than D ∗ ( x + k ˆ u ) , k > , i.e. x is a localmaximum of D ∗ . If x is a local maximum of D ∗ , f ∗ ( k ) musthave a local maximum at k = 0 (the center of each subplot).The result reported below was observed in all 10 different runsof the experiment.Fig. 2 demonstrates the problem on MNIST. The generatorand discriminator are 3 hidden layer MLPs with 512 hiddenneurons. SGD with constant learning rate α = 3 × − wasued in training.As shown in Fig. 2, the generated images keep changingfrom one shape to another, implying that the game does notconverge to an equilibrium. In a large neighborhood aroundevery real image, the discriminator’s output is monotonic inthe sampled direction. At iteration 100000, for every image, f is a decreasing function (Fig. 2f), while at iteration 200000, f is an increasing function (Fig. 2g). More conretely, let ∇ ˆ u D t ( x ) be the discriminator’s directional derivative alongdirection ˆ u at x at iteration t . Then Fig. 2f and 2g shows that ∇ ˆ u D ( x ) and ∇ ˆ u D ( x ) for some x near thereal datapoint x , have opposite directions. The knowledge of D (what D learned on { p g , p r } ) is not usablefor { p g , p r } .We trained DCGAN [20] on CelebA [21] and CIFAR-10[22] to study the effect of network architecture and datasetcomplexity on the level of forgetting. Network architecture andhyper parameters are given in Table II.On CelebA, Fig. 9a - 9g show that CNN suffers less from CFthan MLP. The discriminator in DCGAN-NS is not directionalmonotonic and it successfully makes many real datapoints itslocal maxima (see Sec. IV for more). The discriminator caneffectively discriminate real images from neighboring noisyimages. The generator moves fake datapoints toward theselocal maxima and produces recognizable faces.On CIFAR-10 (Fig. 10a - 10g), the discriminator cannotdiscriminate real images from noisy images. The function f ( k ) in Fig. 10b is almost an increasing function while in Fig. 10d it is almost a decreasing function. The training does not convergeas fake images change significantly as the learning progresses. Conclusion : GAN-NS trained on high dimensional datasetsexhibits the same catastrophic forgetting behaviors as on toydatasets: (1) real datapoints are not local maxima of thediscriminator or in more extreme cases, the discriminator isdirectionally monotonic in the neighborhoods of real datapoints;(2) the gradients w.r.t. datapoints in the neighborhood of a realdatapoint change their directions significantly as fake datapointsmove. 3) The causes of Catastrophic Forgetting: Based on theabove experiments, we identified two reasons for CF:1) Information from previous tasks is not carried to/usedfor the current task. SGD does not use information fromprevious model distributions, p As oldknowledge is overwritten by new knowledge, optimizingthe discriminator on the current task will degrade itsperformance on older tasks.Methods for preventing CF is studied in Section V.IV. T HE OUTPUT LANDSCAPE A. The evolution of the landscape We apply the visualization technique in Section III-B2 toother variants of GAN. We reuse the network architecture andlearning rate from the experiment in Fig. 2. We replace SGDwith Adam with β = 0 . , β = 0 . . We run each experiment10 times with different random seeds and report results thatare consistent between different runs. The evolution of thelandscape and generated samples of GAN-NS, GAN-0GP with λ = 100 , GAN-R1 with λ = 100 , and WGAN-GP with λ = 10 are shown in Fig. 3, 4, 5, and 6 respectively.GAN-0GP, GAN-R1, and WGAN-GP have significantlybetter sample quality and diversity than GAN-NS. GAN-NSdoes not exhibit good convergence behavior: the digit in a imagechanges from one digit to another as the training progresses(Fig. 3). GAN-0GP, GAN-R1, and WGAN-GP exhibit betterconvergence behaviors: for many images, the digits stay thesame during training.We observe that throughout the training process of GAN-0GP,GAN-R1, and WGAN-GP, for every real datapoint, the function f ( k ) always has a local maximum at k = 0 , implying that realdatapoints are local maxima of the discriminator. This can alsobe seen in GAN-R1 trained on the 8 Gaussian dataset (Fig. 1e- 1g): the gradients w.r.t. datapoints in the neighborhood of areal datapoint point toward that real datapoint (GAN-0GP and Note that this does not contradict the statement in [11] that GAN-NSconverge to an equilibrium. Many of the assumptions in that paper is notsatisfied in practice, e.g. the learning rate is not decayed toward 0.a) Real 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (b) Landscape 50000 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (c) Landscape 100000 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (d) Landscape 200000(e) Generated 50000 (f) Generated 100000 (g) Generated 200000 Fig. 2: Catastrophic forgetting problem in GAN-NS trained with SGD. (a) real datapoints from MNIST dataset. (b) - (d) thelandscape around these real datapoints at different training iterations. In each subplot, the X -axis represent k , the Y -axisrepresent D ( · ) . (e) - (g) generated data at different iterations. The same noise inputs were used for all iterations. (a) Real 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (b) Landscape 50000 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (c) Landscape 100000 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (d) Landscape 200000(e) Generated 50000 (f) Generated 100000 (g) Generated 200000 Fig. 3: Output landscape and generated samples from GAN-NS + Adam. (a) Real 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (b) Landscape 50000 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (c) Landscape 100000 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (d) Landscape 200000(e) Generated 50000 (f) Generated 100000 (g) Generated 200000 Fig. 4: Output landscape and generated samples from GAN-0GP with λ = 100 . (a) Real 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (b) Landscape 50000 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (c) Landscape 100000 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (d) Landscape 200000(e) Generated 50000 (f) Generated 100000 (g) Generated 200000 Fig. 5: Output landscape and generated samples from GAN-R1, λ = 100 . a) Real 100 50 0 50 100108642024 100 50 0 50 1001210864202 100 50 0 50 10042024 100 50 0 50 10064202 100 50 0 50 10012108642024 100 50 0 50 1008642024 100 50 0 50 10010.07.55.02.50.02.55.0 100 50 0 50 100864202100 50 0 50 1001086420246 100 50 0 50 1008642024 100 50 0 50 10010.07.55.02.50.02.55.07.5 100 50 0 50 10012108642024 100 50 0 50 100864202 100 50 0 50 10086420246 100 50 0 50 10086420246 100 50 0 50 1008642024100 50 0 50 1008642024 100 50 0 50 1001086420246 100 50 0 50 10064202468 100 50 0 50 10010864202 100 50 0 50 10064202 100 50 0 50 1001086420246 100 50 0 50 10064202 100 50 0 50 100108642024100 50 0 50 1001210864202 100 50 0 50 10086420246 100 50 0 50 100642024 100 50 0 50 1008642024 100 50 0 50 10064202 100 50 0 50 10064202468 100 50 0 50 100864202 100 50 0 50 10010864202100 50 0 50 100108642024 100 50 0 50 100108642024 100 50 0 50 100543210123 100 50 0 50 100108642024 100 50 0 50 100642024 100 50 0 50 10012.510.07.55.02.50.02.55.0 100 50 0 50 1008642024 100 50 0 50 10012.510.07.55.02.50.02.55.0100 50 0 50 100420246 100 50 0 50 10043210123 100 50 0 50 100864202 100 50 0 50 1008642024 100 50 0 50 1001086420 100 50 0 50 10086420246 100 50 0 50 100108642024 100 50 0 50 1001086420100 50 0 50 100642024 100 50 0 50 100642024 100 50 0 50 1001086420 100 50 0 50 10012108642024 100 50 0 50 10012.510.07.55.02.50.02.5 100 50 0 50 10086420246 100 50 0 50 10010864202 100 50 0 50 100864202100 50 0 50 1008642024 100 50 0 50 1006420246 100 50 0 50 1001086420246 100 50 0 50 1001210864202 100 50 0 50 1008642024 100 50 0 50 1006420246 100 50 0 50 100321012345 100 50 0 50 100864202 (b) Landscape 50000 100 50 0 50 1004202468 100 50 0 50 100420246 100 50 0 50 10021012345 100 50 0 50 10042024 100 50 0 50 10042024 100 50 0 50 10021012345 100 50 0 50 10042024 100 50 0 50 10042024100 50 0 50 100420246 100 50 0 50 100642024 100 50 0 50 100420246 100 50 0 50 10043210123 100 50 0 50 10020246 100 50 0 50 10001234567 100 50 0 50 10032101234 100 50 0 50 10032101234100 50 0 50 1006420246 100 50 0 50 100420246 100 50 0 50 100202468 100 50 0 50 100321012345 100 50 0 50 100642024 100 50 0 50 10042024 100 50 0 50 100432101234 100 50 0 50 100420246100 50 0 50 10042024 100 50 0 50 100202468 100 50 0 50 10020246 100 50 0 50 1006420246 100 50 0 50 10042024 100 50 0 50 10010123456 100 50 0 50 100642024 100 50 0 50 10043210123100 50 0 50 10020246 100 50 0 50 10042024 100 50 0 50 1002024 100 50 0 50 1006420246 100 50 0 50 100420246 100 50 0 50 100420246 100 50 0 50 100420246 100 50 0 50 100420246100 50 0 50 10001234567 100 50 0 50 100432101234 100 50 0 50 10042024 100 50 0 50 100420246 100 50 0 50 10032101234 100 50 0 50 10020246 100 50 0 50 10042024 100 50 0 50 10042024100 50 0 50 10032101234 100 50 0 50 10042024 100 50 0 50 10043210123 100 50 0 50 10042024 100 50 0 50 10042024 100 50 0 50 1006420246 100 50 0 50 10032101234 100 50 0 50 10042024100 50 0 50 10020246 100 50 0 50 1004202468 100 50 0 50 100321012345 100 50 0 50 100420246 100 50 0 50 10042024 100 50 0 50 100321012345 100 50 0 50 1004202468 100 50 0 50 100420246 (c) Landscape 100000 100 50 0 50 1000246810 100 50 0 50 10020246 100 50 0 50 10021012345 100 50 0 50 100210123456 100 50 0 50 100101234567 100 50 0 50 10021012345 100 50 0 50 10010123456 100 50 0 50 10010123456100 50 0 50 10020246 100 50 0 50 1000123456 100 50 0 50 100202468 100 50 0 50 10020246 100 50 0 50 1002101234 100 50 0 50 1000246810 100 50 0 50 10010123456 100 50 0 50 10001234567100 50 0 50 1000123456 100 50 0 50 10002468 100 50 0 50 10001234567 100 50 0 50 10010123 100 50 0 50 10010123456 100 50 0 50 10042024 100 50 0 50 1001012345 100 50 0 50 10020246100 50 0 50 1002101234 100 50 0 50 10002468 100 50 0 50 10001234567 100 50 0 50 10020246 100 50 0 50 1001012345 100 50 0 50 1000246810 100 50 0 50 1003210123 100 50 0 50 10021012345100 50 0 50 10002468 100 50 0 50 10001234567 100 50 0 50 10002468 100 50 0 50 10001234567 100 50 0 50 10020246 100 50 0 50 100202468 100 50 0 50 10001234567 100 50 0 50 10002468100 50 0 50 10001234567 100 50 0 50 100101234567 100 50 0 50 100101234567 100 50 0 50 100210123456 100 50 0 50 10020246 100 50 0 50 10020246 100 50 0 50 100210123456 100 50 0 50 10021012345100 50 0 50 100210123456 100 50 0 50 1000123456 100 50 0 50 1001012345 100 50 0 50 1001.00.50.00.51.01.52.02.53.0 100 50 0 50 1000123456 100 50 0 50 10032101234 100 50 0 50 100210123456 100 50 0 50 1000123456100 50 0 50 10020246 100 50 0 50 1000246810 100 50 0 50 100012345 100 50 0 50 1002101234 100 50 0 50 100202468 100 50 0 50 100202468 100 50 0 50 100101234567 100 50 0 50 10010123456 (d) Landscape 200000(e) Generated 50000 (f) Generated 100000 (g) Generated 200000 Fig. 6: Output landscape and generated samples from WGAN-GP, λ = 10 , 5 discriminator updates per 1 generator update.WGANGP exhibit the same behaviors). If a fake datapoint is inthe basin of attraction of a real datapoint and gradient updatesare applied directly on the fake datapoint, it will be attractedtoward the real datapoint. Different attractors (local maxima)at different regions of the data space attract different fakedatapoints toward different directions, spreading fake datapointsover the space, effectively reducing mode collapse.Fig. 7 shows that GAN-0GP with λ = 10 suffers frommild mode collapse. The maxima in Fig. 7 are much sharperthan those in Fig. 6. The discriminator overfits to the realtraining datapoints and forces the scores of near by datapointsto be close to 0. That creates many flat regions where thegradients of the discriminator w.r.t. datapoints in these regionsare vanishingly small. A fake datapoint located in a flat regioncannot move toward the real datapoint because the gradient isvanishingly small. Real datapoints in Fig. 7 have small basinof attraction and cannot effectively spread fake samples overthe space. The diversity of generated samples is thus reduced,making mode collapse visible. In order to attract fake datapointstoward different directions, local maxima should be wide , i.e.they should have large basin of attraction.The landscapes of GAN-NS in Fig. 2 and 3 contain manyflat regions where the scores D ( · ) are very close to 1 or 0. Thesame problem is seen on the 8 Gaussian dataset (datapointsin the orange and blue boxes in Fig. 1a-1d have scores closeto 1 and 0, respectively). However, unlike Fig. 7, the realdatapoints in Fig. 1a - 1d, 2, and 3 are not local maxima. Thediscriminator in GAN-NS underfits the data.CNN based discriminators do not create flat regions in theoutput landscape (Fig. 9b-9d and 10b-10d). However, when thedataset is more complicated, DCGAN-NS discriminator failsto make real datapoints local maxima and the training does notconverge (Fig. 10a-10g). The discriminator underfits the databecause it is not powerful enough to learn features that separatereal and fake/noisy samples. More powerful discriminatorsbased on ResNet [23] significantly improve the quality ofGANs (e.g. [24]). We make the following observation: Observation 2. For a GAN to converge to a good localequilibrium, real datapoints should be wide local maximaof the discriminator. This is consistent with the analysis by the authors of GAN-0GP. Thanh-Tung et al. claimed that larger λ leads to better generalization but may slowdown the training. (a) Generated 100000 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (b) Landscape 100000 Fig. 7: Mode collapse without CF in GAN-0GP, λ = 10 . B. The effect of catastrophic forgetting on the landscape (a) Iter. 0 (b) Iter. 10 (c) Iter. 100 (d) Iter. 200 (e) Iter. 300 (f) Optimal (g) Iter. 0 (h) Iter. 10 (i) Iter. 125 (j) Iter. 250 Fig. 8: High capacity Dirac GAN with n = 2 . The blueline represents the discriminator’s function. The real and fakedatapoints are shown by the blue and red dots, respectively.(a) - (e): Dirac GAN trained on the current fake exampleonly. (f): empirically optimal Dirac discriminator trained onthe current fake example only. (g) - (j): Dirac GAN trained ontwo fake examples: old fake example on the left and currentfake example on the right.We investigate the effect of CF on Dirac GAN [9], a GANthat learns a 1 dimensional Dirac distribution located at theorigin, p r = δ . In the original Dirac GAN, the discriminatoris a linear function with 1 parameter, D ( x ) = ψx, ψ ∈ [ − , and the model distribution is a Dirac distribution located at θ , p g = δ θ . θ is the generator’s parameter. Initially, θ (cid:54) = 0 . Ateach iteration, the training dataset of Dirac GAN contains twotraining examples: a real training example x = 0 , and a fakeraining example y = θ . Gradient updates are applied directlyon the fake training example. − L diracG = L diracD = − D (0) + D ( x ) (3)The unique equilibrium is ψ = θ = 0 . Mescheder et al.showed that the players in Dirac GAN do not converge toan equilibrium (see Fig. 1 in [9]). To make the game convergeto the above equilibrium, the authors proposed R1 gradientpenalty which pushes the gradient w.r.t. the real datapoint to (Table I). A high dimensional GAN can be narrowed to a DiracGAN by considering a pair of real and fake sample and thediscriminator’s output along the line connecting these samples(similar to the landscape in Fig. 2-6).Because the discriminator in the original Dirac GAN is alinear function with a single parameter, the output of Diracdiscriminator is always a monotonic function. We consider ageneric discriminator which is a 1 hidden layer neural network: ˆ D ( x ) = Ψ (cid:62) σ ( Ψ x ) where Ψ , Ψ ∈ [ − , n × , and σ isa monotonically increasing activation function such as LeakyReLU (Fig. 8). At equilibrium, θ = 0 and ˆ D ( x ) is any functionwith a global maximum at x = 0 . Although ˆ D can have globalmaxima (see Fig. 8h), optimizing ˆ D only on the current taskmakes ˆ D a monotonic function (Fig. 8f). Proposition 1. The optimal Dirac discriminator ˆ D ∗ ( x ) thatminimizes L diracD in Eqn. 3 is a monotonic function.Proof. Let ˆ D ( x ) = Ψ (cid:62) σ ( Ψ x ) where Ψ , Ψ ∈ [ − , n × be the discriminator and σ be a non-decreasing activationfunction such as ReLU, Leaky ReLU, Sigmoid, or Tanh. Let x = 0 be the real datapoint, y = θ (cid:54) = 0 be the fake datapoint.The empirically optimal discriminator D ∗ must maximize thedifference D ∗ ( x ) − D ∗ ( y ) . ˆ D ( x ) = Ψ (cid:62) σ ( Ψ × Ψ (cid:62) σ ( )= n (cid:88) i =1 Ψ ,i σ (0)ˆ D ( y ) = Ψ (cid:62) σ ( Ψ × y )= n (cid:88) i =1 Ψ ,i σ (Ψ ,i y )ˆ D ( x ) − ˆ D ( y ) = n (cid:88) i =1 Ψ ,i × ( σ (0) − σ (Ψ ,i y )) Because Ψ ,i y ≤ | y | and σ is non-decreasing σ (0) − σ ( −| y | ) ≥ σ (0) − σ (Ψ ,i y ) ≥ σ (0) − σ ( | y | ) If σ is ReLU or Leaky ReLU or Tanh, then σ (0) = 0 , | σ ( | y | ) | ≥ | σ ( −| y | ) | , thus | σ (0) − σ ( | y | ) | > | σ (0) − σ ( −| y | ) | Architecture DCGAN Pytorch exampleLearning rate 2e-4Batch size 64Optimizer Adam, β = 0 . , β = 0 . No. filters at 1st layer 64 TABLE II: DCGAN model architecture & hyper parameters.If σ is Sigmoid, then σ (0) = 0 . and | σ (0) − σ ( | y | ) | = | σ (0) − σ ( −| y | ) | . For both cases, we have | σ (0) − σ (Ψ ,i y ) | ≤ | σ (0) − σ ( | y | ) | (4)Thus Ψ ,i ( σ (0) − σ (Ψ ,i y )) ≤ × | σ (0) − σ ( | y | ) | (5)The equality for both Eqn. 1 and 2 is achieved for all caseswhen Ψ ,i = − and σ (Ψ ,i y ) = σ ( | y | ) ⇒ Ψ ,i y = | y | ⇒ Ψ ,i = sign ( y ) . The optimal discriminator’s parameters are Ψ ∗ = sign ( y ) × , Ψ ∗ = − . D ( x ) = − (cid:62) σ ( x × sign ( y ) × ) Without loss of generality, assume sign ( y ) = 1 . D ( x ) = − (cid:62) σ ( x × ) = − nσ ( x ) Because σ is monotonic, D ( x ) is monotonic.Optimizing the performance of ˆ D pushes it toward ˆ D ∗ ,making ˆ D monotonic (Fig. 8a - 8e). This explains thedirectional monotonicity of discriminators in Fig. 1a-1d, 2.Although the discriminator in Fig. 8f minimizes the scoreof the current fake datapoint, it assigns high scores to (old)fake datapoints on the left of the real datapoint, i.e. it forgetsthese datapoints. If the discriminator is fixed, then minimizing L diracG corresponds to moving θ to −∞ . Dirac GAN with amonotonic discriminator does not converge. When the generatorand discriminator are trained with alternating SGD, the twoplayers oscillate around the equilibrium (Fig. 8a - 8e).The problem can be alleviated if one old fake datapoint isadded to the training dataset. Fig. 8g - 8j shows that whenold fake example is added, Dirac GAN has better convergencebehavior (the small fluctuation is due to the large constantlearning rate of 0.1). The discriminator at iteration 10 has aglobal maximum at the origin. If the discriminator is fixed,then θ will converge to 0. The experiment suggests thatinformation about previous model distributions helps GANsconverge. [25] used a buffer of recent old fake samples torefine reasonably good fake samples. Recent old fake samplesreduce the oscillation around the equilibrium, helping GANs toconverge faster and produce sharper images. However, becausethe number of samples needed to capture the statistics of adistribution grows exponentially with it dimensionality, storingold fake datapoints is not efficient for high dimensional data. Inthe next section, we study more efficient methods for preservinginformation about old distributions. a) Real 100 50 0 50 1000.000.010.020.030.040.050.06 100 50 0 50 1000.000.020.040.060.08 100 50 0 50 1000.00.10.20.30.40.50.6 100 50 0 50 1000.000.050.100.150.200.25 100 50 0 50 1000.20.30.40.50.60.70.80.91.0 100 50 0 50 1000.000.020.040.060.080.100.12 100 50 0 50 1000.000.010.020.030.040.050.06 100 50 0 50 1000.00.20.40.60.8100 50 0 50 1000.0000.0050.0100.0150.0200.0250.0300.0350.040 100 50 0 50 1000.000.020.040.060.08 100 50 0 50 1000.000.050.100.150.200.250.300.350.40 100 50 0 50 1000.000.050.100.150.200.250.30 100 50 0 50 1000.00.10.20.30.40.50.6 100 50 0 50 1000.00.10.20.30.40.50.6 100 50 0 50 1000.000.010.020.030.040.050.06 100 50 0 50 1000.00.20.40.60.8100 50 0 50 1000.000.010.020.030.040.050.060.070.08 100 50 0 50 1000.000.010.020.030.040.050.060.07 100 50 0 50 1000.0000.0250.0500.0750.1000.1250.1500.175 100 50 0 50 1000.000.010.020.030.040.050.060.07 100 50 0 50 1000.000.050.100.150.200.250.300.35 100 50 0 50 1000.00.20.40.60.8 100 50 0 50 1000.000.020.040.060.080.100.120.14 100 50 0 50 1000.00.10.20.30.4100 50 0 50 1000.000.020.040.060.08 100 50 0 50 1000.000.010.020.030.040.050.060.07 100 50 0 50 1000.00.10.20.30.4 100 50 0 50 1000.20.30.40.50.60.70.80.91.0 100 50 0 50 1000.00.10.20.30.40.5 100 50 0 50 1000.10.20.30.40.50.6 100 50 0 50 1000.000.050.100.150.200.250.300.35 100 50 0 50 1000.0000.0250.0500.0750.1000.1250.1500.175100 50 0 50 1000.00.10.20.30.40.5 100 50 0 50 1000.00.10.20.30.40.50.60.7 100 50 0 50 1000.000.050.100.150.20 100 50 0 50 1000.000.010.020.030.040.05 100 50 0 50 1000.000.010.020.030.040.050.060.07 100 50 0 50 1000.0000.0250.0500.0750.1000.1250.1500.1750.200 100 50 0 50 1000.00.10.20.30.40.5 100 50 0 50 1000.000.050.100.150.20100 50 0 50 1000.0000.0250.0500.0750.1000.1250.1500.1750.200 100 50 0 50 1000.00.10.20.30.40.50.6 100 50 0 50 1000.000.020.040.060.080.100.12 100 50 0 50 1000.00.10.20.30.40.50.60.7 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.10.20.30.40.50.60.7 100 50 0 50 1000.10.20.30.40.50.60.70.80.9 100 50 0 50 1000.000.020.040.060.080.10100 50 0 50 1000.00.10.20.30.40.50.6 100 50 0 50 1000.000.020.040.060.08 100 50 0 50 1000.000.010.020.030.040.05 100 50 0 50 1000.000.020.040.060.080.100.12 100 50 0 50 1000.00.10.20.30.40.50.60.70.8 100 50 0 50 1000.00.10.20.30.4 100 50 0 50 1000.000.050.100.150.200.25 100 50 0 50 1000.000.010.020.030.040.050.060.07100 50 0 50 1000.00.10.20.30.40.50.60.7 100 50 0 50 1000.00.10.20.30.40.50.60.7 100 50 0 50 1000.00.10.20.30.40.5 100 50 0 50 1000.00.20.40.60.8 100 50 0 50 1000.00.10.20.30.40.5 100 50 0 50 1000.00.10.20.30.40.5 100 50 0 50 1000.020.040.060.080.100.12 100 50 0 50 1000.000.050.100.150.200.250.300.35 (b) Land. 5000 100 50 0 50 1000.000.010.020.030.040.050.06 100 50 0 50 1000.000.020.040.060.080.100.12 100 50 0 50 1000.000.050.100.150.200.250.300.35 100 50 0 50 1000.00.10.20.30.40.50.6 100 50 0 50 1000.00.10.20.30.40.50.6 100 50 0 50 1000.0000.0020.0040.0060.008 100 50 0 50 1000.00.10.20.30.40.5 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.0000.0250.0500.0750.1000.1250.1500.175 100 50 0 50 1000.000.050.100.150.200.250.300.350.40 100 50 0 50 1000.00.20.40.60.8 100 50 0 50 1000.000.020.040.060.080.100.12 100 50 0 50 1000.00.10.20.30.40.50.60.7 100 50 0 50 1000.00.10.20.30.40.50.60.70.8 100 50 0 50 1000.00.10.20.30.40.5 100 50 0 50 1000.00.20.40.60.8100 50 0 50 1000.00.10.20.30.40.5 100 50 0 50 1000.000.020.040.060.080.100.120.14 100 50 0 50 1000.000.020.040.060.080.10 100 50 0 50 1000.000.020.040.060.080.10 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.8100 50 0 50 1000.00.20.40.60.8 100 50 0 50 1000.000.050.100.150.200.250.300.350.40 100 50 0 50 1000.00.20.40.60.8 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.10.20.30.40.5 100 50 0 50 1000.00.10.20.30.40.50.60.70.8 100 50 0 50 1000.00.10.20.30.40.50.60.7 100 50 0 50 1000.000.050.100.150.200.25100 50 0 50 1000.00.10.20.30.40.50.60.70.8 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.8 100 50 0 50 1000.000.020.040.060.08 100 50 0 50 1000.00.20.40.60.8 100 50 0 50 1000.00.20.40.60.8 100 50 0 50 1000.0000.0250.0500.0750.1000.1250.1500.175 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.000.020.040.060.080.100.12 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.0000.0250.0500.0750.1000.1250.1500.175 100 50 0 50 1000.00.10.20.30.40.50.60.70.8 100 50 0 50 1000.00.20.40.60.8 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.8 100 50 0 50 1000.00.10.20.30.4100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.10.20.30.40.50.60.70.8 100 50 0 50 1000.000.010.020.030.040.05 100 50 0 50 1000.000.050.100.150.200.250.30 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.000.050.100.150.200.250.30 100 50 0 50 1000.000.010.020.030.04100 50 0 50 1000.00.20.40.60.8 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.8 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.000.050.100.150.200.250.300.35 100 50 0 50 1000.000.010.020.030.040.050.060.070.08 100 50 0 50 1000.00.20.40.60.8 (c) Land. 10000 100 50 0 50 1000.0000.0050.0100.0150.0200.0250.030 100 50 0 50 1000.000.020.040.060.080.10 100 50 0 50 1000.00000.00250.00500.00750.01000.01250.01500.01750.0200 100 50 0 50 1000.000.010.020.030.040.050.06 100 50 0 50 1000.000.050.100.150.200.25 100 50 0 50 1000.0000.0050.0100.0150.0200.0250.030 100 50 0 50 1000.000.020.040.060.080.100.120.14 100 50 0 50 1000.00.10.20.30.40.5100 50 0 50 1000.0000.0050.0100.0150.0200.0250.0300.035 100 50 0 50 1000.000.020.040.060.080.100.120.14 100 50 0 50 1000.000.020.040.060.080.100.12 100 50 0 50 1000.000.020.040.060.08 100 50 0 50 1000.000.020.040.060.080.10 100 50 0 50 1000.000.020.040.060.08 100 50 0 50 1000.0000.0250.0500.0750.1000.1250.1500.175 100 50 0 50 1000.000.050.100.150.200.25100 50 0 50 1000.000.010.020.030.040.050.060.070.08 100 50 0 50 1000.000.020.040.060.080.100.12 100 50 0 50 1000.000.020.040.060.080.100.12 100 50 0 50 1000.000.010.020.030.040.05 100 50 0 50 1000.000.050.100.150.20 100 50 0 50 1000.000.020.040.060.080.100.120.140.16 100 50 0 50 1000.0000.0250.0500.0750.1000.1250.1500.1750.200 100 50 0 50 1000.000.020.040.060.080.100.12100 50 0 50 1000.000.020.040.060.080.100.12 100 50 0 50 1000.000.020.040.060.08 100 50 0 50 1000.000.050.100.150.20 100 50 0 50 1000.000.020.040.060.080.100.120.14 100 50 0 50 1000.000.020.040.060.080.100.120.140.16 100 50 0 50 1000.000.050.100.150.200.250.300.35 100 50 0 50 1000.000.050.100.150.200.250.300.35 100 50 0 50 1000.0000.0250.0500.0750.1000.1250.1500.1750.200100 50 0 50 1000.000.020.040.060.080.100.12 100 50 0 50 1000.000.050.100.150.20 100 50 0 50 1000.00.10.20.30.4 100 50 0 50 1000.000.020.040.060.080.100.12 100 50 0 50 1000.000.010.020.030.040.050.06 100 50 0 50 1000.000.050.100.150.200.25 100 50 0 50 1000.000.010.020.030.04 100 50 0 50 1000.000.050.100.150.200.250.30100 50 0 50 1000.00000.00250.00500.00750.01000.01250.01500.0175 100 50 0 50 1000.000.050.100.150.200.25 100 50 0 50 1000.000.020.040.060.080.100.12 100 50 0 50 1000.000.050.100.150.20 100 50 0 50 1000.000.050.100.150.200.250.300.35 100 50 0 50 1000.000.050.100.150.200.25 100 50 0 50 1000.000.050.100.150.200.25 100 50 0 50 1000.000.020.040.060.080.100.120.140.16100 50 0 50 1000.0000.0250.0500.0750.1000.1250.1500.175 100 50 0 50 1000.000.020.040.060.080.10 100 50 0 50 1000.000.010.020.030.040.050.06 100 50 0 50 1000.0000.0020.0040.0060.0080.0100.0120.014 100 50 0 50 1000.000.020.040.060.080.100.12 100 50 0 50 1000.00.10.20.30.40.5 100 50 0 50 1000.000.020.040.060.080.100.12 100 50 0 50 1000.0000.0020.0040.0060.0080.0100.0120.014100 50 0 50 1000.000.010.020.030.040.050.06 100 50 0 50 1000.00.10.20.30.40.50.6 100 50 0 50 1000.000.020.040.060.080.10 100 50 0 50 1000.00.10.20.30.4 100 50 0 50 1000.000.050.100.150.200.250.300.35 100 50 0 50 1000.000.020.040.060.080.100.120.14 100 50 0 50 1000.0000.0050.0100.0150.0200.0250.0300.035 100 50 0 50 1000.0000.0250.0500.0750.1000.1250.1500.175 (d) Land. 20000(e) Gen. 5000 (f) Gen. 10000 (g) Gen. 20000(h) Real 100 50 0 50 1000.000.020.040.060.080.10 100 50 0 50 1000.0000.0050.0100.0150.0200.0250.0300.035 100 50 0 50 1000.0050.0100.0150.0200.0250.030 100 50 0 50 1000.00000.00250.00500.00750.01000.01250.01500.0175 100 50 0 50 1000.0000.0020.0040.0060.0080.0100.0120.0140.016 100 50 0 50 1000.0000.0010.0020.0030.0040.0050.006 100 50 0 50 1000.0000.0020.0040.0060.008 100 50 0 50 1000.000.020.040.060.080.100.120.14100 50 0 50 1000.000.010.020.030.040.050.060.070.08 100 50 0 50 1000.000.050.100.150.200.25 100 50 0 50 1000.000.010.020.030.04 100 50 0 50 1000.0000.0050.0100.0150.0200.025 100 50 0 50 1000.000.010.020.030.040.050.06 100 50 0 50 1000.0000.0020.0040.0060.0080.0100.012 100 50 0 50 1000.0000.0010.0020.0030.0040.005 100 50 0 50 1000.0000.0020.0040.0060.008100 50 0 50 1000.10.20.30.40.50.60.70.8 100 50 0 50 1000.0000.0050.0100.0150.020 100 50 0 50 1000.00000.00250.00500.00750.01000.01250.01500.0175 100 50 0 50 1000.0010.0020.0030.0040.0050.006 100 50 0 50 1000.000.010.020.030.040.05 100 50 0 50 1000.000.020.040.060.080.10 100 50 0 50 1000.000.020.040.060.080.100.120.140.16 100 50 0 50 1000.0000.0010.0020.0030.0040.005100 50 0 50 1000.000.010.020.030.040.050.060.070.08 100 50 0 50 1000.00000.00250.00500.00750.01000.01250.01500.01750.0200 100 50 0 50 1000.0000.0020.0040.0060.008 100 50 0 50 1000.0000.0010.0020.0030.0040.0050.006 100 50 0 50 1000.0000.0020.0040.0060.0080.010 100 50 0 50 1000.0050.0100.0150.0200.025 100 50 0 50 1000.0020.0040.0060.0080.010 100 50 0 50 1000.00.10.20.30.40.50.60.7100 50 0 50 1000.0000.0020.0040.0060.0080.010 100 50 0 50 1000.010.020.030.040.050.060.07 100 50 0 50 1000.00000.00250.00500.00750.01000.01250.01500.0175 100 50 0 50 1000.0000.0050.0100.0150.020 100 50 0 50 1000.010.020.030.04 100 50 0 50 1000.0000.0050.0100.0150.0200.0250.030 100 50 0 50 1000.0050.0100.0150.0200.025 100 50 0 50 1000.00250.00500.00750.01000.01250.01500.01750.0200100 50 0 50 1000.00000.00250.00500.00750.01000.01250.01500.0175 100 50 0 50 1000.0000.0050.0100.0150.0200.0250.0300.0350.040 100 50 0 50 1000.0050.0100.0150.0200.0250.030 100 50 0 50 1000.0000.0050.0100.0150.0200.025 100 50 0 50 1000.0000.0250.0500.0750.1000.1250.1500.1750.200 100 50 0 50 1000.0000.0050.0100.0150.0200.0250.030 100 50 0 50 1000.000.050.100.150.200.25 100 50 0 50 1000.0000.0050.0100.0150.0200.0250.0300.0350.040100 50 0 50 1000.00000.00250.00500.00750.01000.01250.01500.0175 100 50 0 50 1000.010.020.030.040.050.060.07 100 50 0 50 1000.000.020.040.060.08 100 50 0 50 1000.000.020.040.060.080.100.12 100 50 0 50 1000.010.020.030.040.050.06 100 50 0 50 1000.0000.0020.0040.0060.0080.0100.0120.014 100 50 0 50 1000.0000.0050.0100.0150.0200.0250.0300.035 100 50 0 50 1000.000.020.040.060.080.100.120.140.16100 50 0 50 1000.0050.0100.0150.0200.0250.0300.035 100 50 0 50 1000.000.020.040.060.080.10 100 50 0 50 1000.0000.0020.0040.0060.0080.0100.0120.0140.016 100 50 0 50 1000.000.020.040.060.080.10 100 50 0 50 1000.010.020.030.040.050.060.070.08 100 50 0 50 1000.000.020.040.060.080.100.12 100 50 0 50 1000.0000.0050.0100.0150.0200.025 100 50 0 50 1000.0000.0050.0100.0150.020 (i) Land. 5000 100 50 0 50 1000.00000.00250.00500.00750.01000.01250.01500.0175 100 50 0 50 1000.0000.0050.0100.0150.0200.0250.030 100 50 0 50 1000.0020.0040.0060.0080.010 100 50 0 50 1000.0010.0020.0030.0040.0050.006 100 50 0 50 1000.0000.0050.0100.0150.0200.025 100 50 0 50 1000.0000.0050.0100.0150.0200.025 100 50 0 50 1000.00000.00250.00500.00750.01000.01250.01500.0175 100 50 0 50 1000.020.040.060.080.10100 50 0 50 1000.0000.0050.0100.0150.0200.0250.030 100 50 0 50 1000.000.020.040.060.080.100.120.14 100 50 0 50 1000.0000.0050.0100.0150.0200.0250.030 100 50 0 50 1000.000.010.020.030.040.05 100 50 0 50 1000.0000.0010.0020.0030.0040.0050.0060.007 100 50 0 50 1000.00000.00250.00500.00750.01000.01250.01500.0175 100 50 0 50 1000.00000.00250.00500.00750.01000.01250.01500.01750.0200 100 50 0 50 1000.0000.0020.0040.0060.0080.010100 50 0 50 1000.0000.0050.0100.0150.020 100 50 0 50 1000.000.010.020.030.040.05 100 50 0 50 1000.0000.0250.0500.0750.1000.1250.1500.175 100 50 0 50 1000.000.010.020.030.040.050.06 100 50 0 50 1000.000.010.020.030.04 100 50 0 50 1000.0000.0050.0100.0150.020 100 50 0 50 1000.0020.0040.0060.008 100 50 0 50 1000.00000.00250.00500.00750.01000.01250.01500.01750.0200100 50 0 50 1000.0000.0050.0100.0150.0200.025 100 50 0 50 1000.0000.0050.0100.0150.0200.0250.030 100 50 0 50 1000.0000.0250.0500.0750.1000.1250.1500.1750.200 100 50 0 50 1000.0000.0050.0100.0150.020 100 50 0 50 1000.0020.0040.0060.0080.010 100 50 0 50 1000.00.10.20.30.4 100 50 0 50 1000.0000.0050.0100.0150.0200.0250.030 100 50 0 50 1000.000.020.040.060.080.100.120.140.16100 50 0 50 1000.0020.0040.0060.0080.010 100 50 0 50 1000.000.050.100.150.200.25 100 50 0 50 1000.0000.0050.0100.0150.0200.0250.030 100 50 0 50 1000.0020.0040.0060.0080.0100.012 100 50 0 50 1000.0000.0050.0100.0150.0200.0250.0300.0350.040 100 50 0 50 1000.0020.0040.0060.0080.0100.0120.0140.0160.018 100 50 0 50 1000.0050.0100.0150.0200.0250.030 100 50 0 50 1000.0000.0010.0020.0030.0040.0050.006100 50 0 50 1000.0010.0020.0030.0040.0050.0060.0070.0080.009 100 50 0 50 1000.000.020.040.060.080.100.12 100 50 0 50 1000.000.020.040.060.080.100.120.140.16 100 50 0 50 1000.0050.0100.0150.020 100 50 0 50 1000.000.010.020.030.04 100 50 0 50 1000.0000.0050.0100.0150.0200.0250.030 100 50 0 50 1000.0040.0060.0080.0100.0120.0140.0160.0180.020 100 50 0 50 1000.0000.0050.0100.0150.0200.025100 50 0 50 1000.0010.0020.0030.0040.0050.006 100 50 0 50 1000.000.010.020.030.04 100 50 0 50 1000.000.020.040.060.08 100 50 0 50 1000.000.010.020.030.040.050.06 100 50 0 50 1000.0000.0050.0100.0150.0200.025 100 50 0 50 1000.0000.0050.0100.0150.0200.0250.030 100 50 0 50 1000.0000.0050.0100.0150.0200.0250.0300.0350.040 100 50 0 50 1000.0000.0250.0500.0750.1000.1250.1500.1750.200100 50 0 50 1000.000.020.040.060.080.100.120.14 100 50 0 50 1000.00000.00250.00500.00750.01000.01250.01500.01750.0200 100 50 0 50 1000.0020.0040.0060.0080.0100.012 100 50 0 50 1000.0050.0100.0150.0200.0250.030 100 50 0 50 1000.0050.0100.0150.0200.025 100 50 0 50 1000.0000.0050.0100.0150.0200.025 100 50 0 50 1000.0000.0250.0500.0750.1000.1250.1500.175 100 50 0 50 1000.0000.0050.0100.0150.0200.0250.0300.0350.040 (j) Land. 10000 100 50 0 50 1000.000.050.100.150.200.250.300.35 100 50 0 50 1000.0000.0250.0500.0750.1000.1250.1500.175 100 50 0 50 1000.000.050.100.150.200.25 100 50 0 50 1000.00.10.20.30.40.50.60.7 100 50 0 50 1000.000.050.100.150.20 100 50 0 50 1000.000.020.040.060.080.100.12 100 50 0 50 1000.0000.0050.0100.0150.020 100 50 0 50 1000.000.050.100.150.200.250.30100 50 0 50 1000.00.10.20.30.40.50.6 100 50 0 50 1000.000.020.040.060.080.10 100 50 0 50 1000.0000.0250.0500.0750.1000.1250.1500.175 100 50 0 50 1000.000.050.100.150.200.250.300.350.40 100 50 0 50 1000.0000.0250.0500.0750.1000.1250.1500.175 100 50 0 50 1000.00.10.20.30.40.5 100 50 0 50 1000.0000.0250.0500.0750.1000.1250.1500.1750.200 100 50 0 50 1000.0000.0050.0100.0150.0200.0250.0300.0350.040100 50 0 50 1000.00.10.20.30.40.5 100 50 0 50 1000.00.20.40.60.8 100 50 0 50 1000.00.20.40.60.8 100 50 0 50 1000.00.10.20.30.40.50.6 100 50 0 50 1000.00.10.20.30.40.50.60.70.8 100 50 0 50 1000.000.010.020.030.040.050.06 100 50 0 50 1000.000.020.040.060.080.10 100 50 0 50 1000.000.050.100.150.200.250.300.350.40100 50 0 50 1000.000.020.040.060.080.10 100 50 0 50 1000.000.020.040.060.080.10 100 50 0 50 1000.000.020.040.060.080.100.120.14 100 50 0 50 1000.00.10.20.30.4 100 50 0 50 1000.000.050.100.150.20 100 50 0 50 1000.00.10.20.30.40.50.60.7 100 50 0 50 1000.00.10.20.30.40.50.60.70.8 100 50 0 50 1000.00.20.40.60.8100 50 0 50 1000.00.10.20.30.4 100 50 0 50 1000.00.10.20.30.40.50.60.70.8 100 50 0 50 1000.00.10.20.30.40.50.6 100 50 0 50 1000.000.020.040.060.080.100.120.140.16 100 50 0 50 1000.00.10.20.30.40.5 100 50 0 50 1000.00.10.20.30.4 100 50 0 50 1000.000.050.100.150.200.25 100 50 0 50 1000.0000.0250.0500.0750.1000.1250.1500.175100 50 0 50 1000.000.020.040.060.080.100.120.140.16 100 50 0 50 1000.00.10.20.30.40.50.6 100 50 0 50 1000.00.10.20.30.40.50.60.70.8 100 50 0 50 1000.00.10.20.30.4 100 50 0 50 1000.00.10.20.30.4 100 50 0 50 1000.00.10.20.30.4 100 50 0 50 1000.00.10.20.30.4 100 50 0 50 1000.00.10.20.30.4100 50 0 50 1000.00.10.20.30.40.50.60.7 100 50 0 50 1000.00.10.20.30.40.5 100 50 0 50 1000.00.10.20.30.40.50.6 100 50 0 50 1000.000.050.100.150.200.250.300.350.40 100 50 0 50 1000.000.050.100.150.200.25 100 50 0 50 1000.00.10.20.30.40.5 100 50 0 50 1000.000.050.100.150.20 100 50 0 50 1000.00.10.20.30.40.5100 50 0 50 1000.000.050.100.150.200.250.300.350.40 100 50 0 50 1000.000.010.020.030.040.050.060.07 100 50 0 50 1000.000.020.040.060.080.10 100 50 0 50 1000.0000.0250.0500.0750.1000.1250.1500.1750.200 100 50 0 50 1000.00.10.20.30.40.50.6 100 50 0 50 1000.00.20.40.60.8 100 50 0 50 1000.00.20.40.60.8 100 50 0 50 1000.00.10.20.30.40.5 (k) Land. 20000(l) Gen. 5000 (m) Gen. 10000 (n) Gen. 20000 Fig. 9: Result on CelebA. (a) - (g) DCGAN-NS. (h) - (n)DCGAN-0GP (a) Real 100 50 0 50 1000.000.020.040.060.080.100.12 100 50 0 50 1000.000.010.020.030.040.05 100 50 0 50 1000.010.020.030.040.050.06 100 50 0 50 1000.010.020.030.040.05 100 50 0 50 1000.010.020.030.040.050.06 100 50 0 50 1000.000.010.020.030.040.050.060.07 100 50 0 50 1000.010.020.030.040.050.060.07 100 50 0 50 1000.000.020.040.060.080.10100 50 0 50 1000.010.020.030.040.050.060.070.080.09 100 50 0 50 1000.0050.0100.0150.0200.0250.0300.0350.040 100 50 0 50 1000.000.020.040.060.080.10 100 50 0 50 1000.0020.0040.0060.0080.0100.012 100 50 0 50 1000.0050.0100.0150.0200.0250.030 100 50 0 50 1000.010.020.030.040.050.06 100 50 0 50 1000.000.020.040.060.080.100.120.14 100 50 0 50 1000.000.010.020.030.04100 50 0 50 1000.0250.0500.0750.1000.1250.1500.1750.200 100 50 0 50 1000.0050.0100.0150.0200.0250.030 100 50 0 50 1000.060.080.100.120.140.16 100 50 0 50 1000.010.020.030.040.05 100 50 0 50 1000.00000.00250.00500.00750.01000.01250.01500.01750.0200 100 50 0 50 1000.000.010.020.030.040.05 100 50 0 50 1000.000.020.040.060.080.100.12 100 50 0 50 1000.0050.0100.0150.0200.0250.0300.0350.040100 50 0 50 1000.020.030.040.050.060.070.080.090.10 100 50 0 50 1000.0020.0040.0060.0080.0100.0120.0140.016 100 50 0 50 1000.020.040.060.080.100.120.14 100 50 0 50 1000.000.020.040.060.08 100 50 0 50 1000.000.010.020.030.040.050.060.07 100 50 0 50 1000.010.020.030.040.05 100 50 0 50 1000.010.020.030.040.050.06 100 50 0 50 1000.0050.0100.0150.0200.0250.030100 50 0 50 1000.0000.0050.0100.0150.0200.0250.030 100 50 0 50 1000.000.010.020.030.040.050.06 100 50 0 50 1000.010.020.030.040.05 100 50 0 50 1000.0000.0050.0100.0150.0200.0250.0300.0350.040 100 50 0 50 1000.010.020.030.040.050.060.07 100 50 0 50 1000.00250.00500.00750.01000.01250.01500.01750.0200 100 50 0 50 1000.0050.0100.0150.0200.0250.0300.0350.040 100 50 0 50 1000.0000.0050.0100.0150.0200.0250.0300.035100 50 0 50 1000.0100.0150.0200.0250.0300.0350.040 100 50 0 50 1000.010.020.030.040.050.060.070.08 100 50 0 50 1000.000.020.040.060.080.10 100 50 0 50 1000.000.010.020.030.040.050.060.07 100 50 0 50 1000.0250.0500.0750.1000.1250.1500.1750.200 100 50 0 50 1000.020.030.040.050.060.070.080.09 100 50 0 50 1000.010.020.030.040.050.06 100 50 0 50 1000.000.010.020.030.04100 50 0 50 1000.000.020.040.060.080.100.120.140.16 100 50 0 50 1000.0050.0100.0150.0200.0250.0300.0350.040 100 50 0 50 1000.040.060.080.100.120.14 100 50 0 50 1000.020.030.040.050.060.070.080.090.10 100 50 0 50 1000.0050.0100.0150.0200.025 100 50 0 50 1000.010.020.030.040.050.060.07 100 50 0 50 1000.000.010.020.030.04 100 50 0 50 1000.000.020.040.060.080.100.120.14100 50 0 50 1000.010.020.030.040.050.060.07 100 50 0 50 1000.000.010.020.030.040.050.06 100 50 0 50 1000.010.020.030.040.050.060.07 100 50 0 50 1000.010.020.030.040.05 100 50 0 50 1000.000.020.040.060.080.10 100 50 0 50 1000.010.020.030.040.05 100 50 0 50 1000.010.020.030.040.050.060.07 100 50 0 50 1000.010.020.030.040.050.060.07 (b) Land. 5000 100 50 0 50 1000.000000.000020.000040.000060.000080.000100.000120.00014 100 50 0 50 1000.000010.000020.000030.000040.000050.000060.000070.000080.00009 100 50 0 50 1000.00000.00050.00100.00150.0020 100 50 0 50 1000.000000.000050.000100.000150.000200.000250.00030 100 50 0 50 1000.00000.00020.00040.00060.0008 100 50 0 50 1000.00000.00010.00020.00030.00040.00050.00060.00070.0008 100 50 0 50 1000.00000.00010.00020.00030.00040.00050.0006 100 50 0 50 1000.00000.00010.00020.00030.0004100 50 0 50 1000.000000.000050.000100.000150.000200.000250.00030 100 50 0 50 1000.000000.000050.000100.000150.000200.000250.000300.00035 100 50 0 50 1000.00000.00010.00020.00030.00040.0005 100 50 0 50 1000.0000000.0000050.0000100.0000150.0000200.0000250.0000300.000035 100 50 0 50 1000.000020.000040.000060.000080.000100.000120.000140.00016 100 50 0 50 1000.000020.000040.000060.000080.000100.000120.000140.00016 100 50 0 50 1000.00000.00020.00040.00060.00080.0010 100 50 0 50 1000.000000.000050.000100.000150.000200.000250.00030100 50 0 50 1000.000050.000100.000150.00020 100 50 0 50 1000.000000.000020.000040.000060.00008 100 50 0 50 1000.0000000.0000250.0000500.0000750.0001000.0001250.0001500.000175 100 50 0 50 1000.000000.000050.000100.000150.000200.000250.000300.000350.00040 100 50 0 50 1000.00000.00020.00040.00060.00080.00100.0012 100 50 0 50 1000.000000.000050.000100.000150.00020 100 50 0 50 1000.000000.000050.000100.000150.000200.000250.000300.000350.00040 100 50 0 50 1000.00000.00010.00020.00030.00040.0005100 50 0 50 1000.00000.00020.00040.00060.00080.00100.0012 100 50 0 50 1000.000020.000040.000060.000080.000100.000120.00014 100 50 0 50 1000.00000.00010.00020.00030.00040.00050.0006 100 50 0 50 1000.000010.000020.000030.000040.000050.000060.000070.00008 100 50 0 50 1000.000000.000020.000040.000060.000080.000100.000120.00014 100 50 0 50 1000.0000000.0000250.0000500.0000750.0001000.0001250.0001500.000175 100 50 0 50 1000.00000.00010.00020.00030.00040.0005 100 50 0 50 1000.00000.00020.00040.00060.00080.00100.00120.00140.0016100 50 0 50 1000.00000.00020.00040.00060.00080.00100.0012 100 50 0 50 1000.00000.00010.00020.00030.00040.00050.0006 100 50 0 50 1000.000000.000250.000500.000750.001000.001250.001500.001750.00200 100 50 0 50 1000.0000000.0000250.0000500.0000750.0001000.0001250.0001500.0001750.000200 100 50 0 50 1000.000000.000050.000100.000150.000200.000250.00030 100 50 0 50 1000.00000.00010.00020.00030.0004 100 50 0 50 1000.00000.00050.00100.00150.0020 100 50 0 50 1000.00000.00020.00040.00060.00080.00100.00120.0014100 50 0 50 1000.00000.00020.00040.00060.00080.00100.0012 100 50 0 50 1000.00000.00020.00040.00060.00080.0010 100 50 0 50 1000.000000.000050.000100.000150.000200.00025 100 50 0 50 1000.000000.000020.000040.000060.000080.00010 100 50 0 50 1000.000250.000500.000750.001000.001250.001500.00175 100 50 0 50 1000.00020.00040.00060.00080.00100.00120.0014 100 50 0 50 1000.0000000.0000250.0000500.0000750.0001000.0001250.0001500.0001750.000200 100 50 0 50 1000.00000.00050.00100.00150.00200.00250.00300.0035100 50 0 50 1000.00000.00010.00020.00030.00040.0005 100 50 0 50 1000.00010.00020.00030.00040.00050.00060.00070.0008 100 50 0 50 1000.000050.000100.000150.00020 100 50 0 50 1000.0000000.0000250.0000500.0000750.0001000.0001250.0001500.000175 100 50 0 50 1000.000010.000020.000030.000040.000050.000060.000070.00008 100 50 0 50 1000.000050.000100.000150.000200.000250.00030 100 50 0 50 1000.00000.00020.00040.00060.00080.0010 100 50 0 50 1000.0000000.0000250.0000500.0000750.0001000.0001250.0001500.0001750.000200100 50 0 50 1000.00000.00010.00020.00030.00040.0005 100 50 0 50 1000.00000.00010.00020.00030.00040.00050.00060.0007 100 50 0 50 1000.000000.000050.000100.000150.000200.000250.000300.00035 100 50 0 50 1000.000000.000020.000040.000060.000080.000100.000120.000140.00016 100 50 0 50 1000.00000.00020.00040.00060.0008 100 50 0 50 1000.00000.00020.00040.00060.00080.00100.0012 100 50 0 50 1000.00000.00020.00040.00060.00080.00100.0012 100 50 0 50 1000.00000.00010.00020.00030.0004 (c) Land. 10000 100 50 0 50 1000.000000.000250.000500.000750.001000.001250.001500.00175 100 50 0 50 1000.000.020.040.060.08 100 50 0 50 1000.0000.0020.0040.0060.0080.0100.012 100 50 0 50 1000.0000.0010.0020.0030.0040.0050.0060.0070.008 100 50 0 50 1000.0000.0010.0020.0030.0040.005 100 50 0 50 1000.0000.0050.0100.0150.020 100 50 0 50 1000.0020.0040.0060.0080.010 100 50 0 50 1000.0000.0050.0100.0150.0200.0250.0300.0350.040100 50 0 50 1000.0050.0100.0150.0200.0250.030 100 50 0 50 1000.00050.00100.00150.00200.00250.0030 100 50 0 50 1000.0000.0010.0020.0030.0040.0050.0060.007 100 50 0 50 1000.0000.0010.0020.0030.0040.0050.0060.007 100 50 0 50 1000.0000.0050.0100.0150.0200.0250.0300.035 100 50 0 50 1000.0000.0020.0040.0060.0080.010 100 50 0 50 1000.000.010.020.030.040.05 100 50 0 50 1000.000.020.040.060.080.100.120.14100 50 0 50 1000.0020.0040.0060.0080.010 100 50 0 50 1000.0000.0050.0100.0150.0200.025 100 50 0 50 1000.00050.00100.00150.0020 100 50 0 50 1000.0020.0040.0060.008 100 50 0 50 1000.0000.0020.0040.0060.0080.0100.0120.014 100 50 0 50 1000.0000.0010.0020.0030.0040.005 100 50 0 50 1000.0000.0010.0020.0030.0040.005 100 50 0 50 1000.0010.0020.0030.0040.0050.0060.0070.008100 50 0 50 1000.00000.00250.00500.00750.01000.01250.01500.0175 100 50 0 50 1000.0000.0050.0100.0150.0200.0250.030 100 50 0 50 1000.0000.0050.0100.0150.0200.0250.0300.035 100 50 0 50 1000.00000.00250.00500.00750.01000.01250.01500.01750.0200 100 50 0 50 1000.0000.0020.0040.0060.0080.0100.0120.014 100 50 0 50 1000.0020.0040.0060.0080.010 100 50 0 50 1000.0040.0060.0080.0100.0120.014 100 50 0 50 1000.000.010.020.030.040.050.06100 50 0 50 1000.00000.00250.00500.00750.01000.01250.01500.0175 100 50 0 50 1000.00000.00250.00500.00750.01000.01250.01500.0175 100 50 0 50 1000.00000.00250.00500.00750.01000.01250.01500.0175 100 50 0 50 1000.0000.0020.0040.0060.0080.0100.0120.0140.016 100 50 0 50 1000.0000.0050.0100.0150.0200.025 100 50 0 50 1000.0000.0020.0040.0060.0080.0100.012 100 50 0 50 1000.00250.00500.00750.01000.01250.01500.0175 100 50 0 50 1000.0000.0010.0020.0030.004100 50 0 50 1000.0000.0050.0100.0150.020 100 50 0 50 1000.00000.00050.00100.00150.00200.00250.0030 100 50 0 50 1000.0020.0040.0060.0080.0100.0120.014 100 50 0 50 1000.0000.0020.0040.0060.008 100 50 0 50 1000.0050.0100.0150.0200.0250.0300.0350.040 100 50 0 50 1000.00000.00250.00500.00750.01000.01250.01500.01750.0200 100 50 0 50 1000.0000.0010.0020.0030.0040.0050.0060.007 100 50 0 50 1000.000.010.020.030.04100 50 0 50 1000.0000.0010.0020.0030.0040.005 100 50 0 50 1000.00000.00050.00100.00150.00200.0025 100 50 0 50 1000.0000.0020.0040.0060.0080.0100.012 100 50 0 50 1000.0010.0020.0030.0040.0050.0060.0070.008 100 50 0 50 1000.0050.0100.0150.0200.0250.0300.0350.040 100 50 0 50 1000.0000.0050.0100.0150.0200.0250.0300.0350.040 100 50 0 50 1000.0000.0020.0040.0060.008 100 50 0 50 1000.0010.0020.0030.0040.0050.0060.0070.008100 50 0 50 1000.00050.00100.00150.00200.00250.00300.00350.0040 100 50 0 50 1000.000.010.020.030.040.050.06 100 50 0 50 1000.0000.0020.0040.0060.0080.0100.0120.014 100 50 0 50 1000.0000.0020.0040.0060.0080.0100.012 100 50 0 50 1000.00000.00250.00500.00750.01000.01250.01500.0175 100 50 0 50 1000.00000.00050.00100.00150.00200.00250.00300.00350.0040 100 50 0 50 1000.00000.00250.00500.00750.01000.01250.01500.01750.0200 100 50 0 50 1000.0000.0020.0040.0060.0080.0100.012 (d) Land. 20000(e) Gen. 5000 (f) Gen. 10000 (g) Gen. 20000(h) Real 100 50 0 50 1000.00000.00050.00100.00150.00200.00250.00300.0035 100 50 0 50 1000.0000.0010.0020.0030.0040.0050.0060.007 100 50 0 50 1000.00000.00050.00100.00150.00200.0025 100 50 0 50 1000.0000.0010.0020.0030.0040.0050.006 100 50 0 50 1000.0000.0010.0020.0030.004 100 50 0 50 1000.0000.0010.0020.0030.004 100 50 0 50 1000.0000.0010.0020.0030.0040.0050.0060.0070.008 100 50 0 50 1000.00000.00050.00100.00150.0020100 50 0 50 1000.000000.000250.000500.000750.001000.001250.001500.00175 100 50 0 50 1000.00000.00050.00100.00150.00200.00250.0030 100 50 0 50 1000.0000.0020.0040.0060.0080.0100.0120.0140.016 100 50 0 50 1000.0000.0010.0020.0030.004 100 50 0 50 1000.0000.0020.0040.0060.0080.010 100 50 0 50 1000.00000.00050.00100.00150.00200.00250.0030 100 50 0 50 1000.0000.0020.0040.0060.008 100 50 0 50 1000.00000.00010.00020.00030.00040.00050.0006100 50 0 50 1000.0000.0020.0040.0060.008 100 50 0 50 1000.0000.0020.0040.0060.0080.0100.012 100 50 0 50 1000.00000.00050.00100.00150.0020 100 50 0 50 1000.00000.00020.00040.00060.00080.00100.00120.0014 100 50 0 50 1000.0000.0020.0040.0060.008 100 50 0 50 1000.0000.0020.0040.0060.0080.010 100 50 0 50 1000.00000.00050.00100.00150.0020 100 50 0 50 1000.00000.00050.00100.00150.0020100 50 0 50 1000.0000.0010.0020.0030.004 100 50 0 50 1000.0000.0010.0020.0030.0040.005 100 50 0 50 1000.00000.00020.00040.00060.00080.00100.00120.00140.0016 100 50 0 50 1000.0000.0020.0040.0060.008 100 50 0 50 1000.0000.0010.0020.0030.0040.005 100 50 0 50 1000.0000.0010.0020.0030.0040.0050.006 100 50 0 50 1000.000000.000250.000500.000750.001000.001250.001500.001750.00200 100 50 0 50 1000.0000.0010.0020.0030.0040.005100 50 0 50 1000.0000.0010.0020.0030.0040.0050.0060.0070.008 100 50 0 50 1000.00000.00050.00100.00150.00200.00250.0030 100 50 0 50 1000.0000.0020.0040.0060.0080.010 100 50 0 50 1000.0000.0010.0020.0030.0040.0050.0060.007 100 50 0 50 1000.0000.0010.0020.0030.004 100 50 0 50 1000.0000.0010.0020.0030.0040.0050.0060.0070.008 100 50 0 50 1000.00000.00050.00100.00150.00200.00250.00300.0035 100 50 0 50 1000.0000.0020.0040.0060.0080.010100 50 0 50 1000.00000.00050.00100.00150.00200.00250.0030 100 50 0 50 1000.00020.00040.00060.00080.00100.00120.00140.0016 100 50 0 50 1000.00000.00050.00100.00150.00200.00250.00300.0035 100 50 0 50 1000.0000.0010.0020.0030.004 100 50 0 50 1000.00000.00050.00100.00150.00200.0025 100 50 0 50 1000.0000.0020.0040.0060.0080.0100.0120.0140.016 100 50 0 50 1000.00000.00050.00100.00150.0020 100 50 0 50 1000.0000.0010.0020.0030.0040.0050.006100 50 0 50 1000.00000.00050.00100.00150.0020 100 50 0 50 1000.0000.0010.0020.0030.0040.005 100 50 0 50 1000.0000.0010.0020.0030.0040.005 100 50 0 50 1000.0000.0010.0020.0030.0040.005 100 50 0 50 1000.0000.0020.0040.0060.0080.010 100 50 0 50 1000.0000.0020.0040.0060.0080.010 100 50 0 50 1000.0000.0010.0020.0030.0040.0050.006 100 50 0 50 1000.00000.00250.00500.00750.01000.01250.01500.0175100 50 0 50 1000.0000.0010.0020.0030.0040.005 100 50 0 50 1000.0000.0010.0020.0030.004 100 50 0 50 1000.0000.0010.0020.0030.0040.005 100 50 0 50 1000.0000.0010.0020.0030.0040.0050.006 100 50 0 50 1000.0000.0010.0020.0030.0040.005 100 50 0 50 1000.0000.0010.0020.0030.0040.005 100 50 0 50 1000.000000.000250.000500.000750.001000.001250.001500.001750.00200 100 50 0 50 1000.0000.0010.0020.0030.0040.0050.006 (i) Land. 5000 100 50 0 50 1000.0000.0250.0500.0750.1000.1250.1500.1750.200 100 50 0 50 1000.050.100.150.200.25 100 50 0 50 1000.020.040.060.080.100.12 100 50 0 50 1000.000.020.040.060.080.100.12 100 50 0 50 1000.000.010.020.030.040.05 100 50 0 50 1000.020.040.060.080.10 100 50 0 50 1000.010.020.030.040.050.060.070.08 100 50 0 50 1000.0050.0100.0150.0200.0250.0300.0350.0400.045100 50 0 50 1000.0250.0500.0750.1000.1250.1500.175 100 50 0 50 1000.000.020.040.060.080.100.120.14 100 50 0 50 1000.0000.0250.0500.0750.1000.1250.1500.1750.200 100 50 0 50 1000.020.040.060.080.100.120.14 100 50 0 50 1000.020.040.060.080.100.120.140.16 100 50 0 50 1000.000.010.020.030.040.050.060.07 100 50 0 50 1000.000.050.100.150.200.250.30 100 50 0 50 1000.000.050.100.150.200.250.30100 50 0 50 1000.000.020.040.060.080.100.120.140.16 100 50 0 50 1000.0000.0250.0500.0750.1000.1250.1500.1750.200 100 50 0 50 1000.050.100.150.200.250.300.35 100 50 0 50 1000.000.050.100.150.200.250.300.35 100 50 0 50 1000.0000.0050.0100.0150.0200.0250.0300.0350.040 100 50 0 50 1000.000.050.100.150.200.25 100 50 0 50 1000.020.040.060.080.100.120.140.160.18 100 50 0 50 1000.000.010.020.030.040.050.060.07100 50 0 50 1000.000.010.020.030.040.050.060.070.08 100 50 0 50 1000.020.040.060.080.100.12 100 50 0 50 1000.000.020.040.060.080.100.120.140.16 100 50 0 50 1000.0000.0250.0500.0750.1000.1250.1500.1750.200 100 50 0 50 1000.000.020.040.060.080.10 100 50 0 50 1000.00.10.20.30.4 100 50 0 50 1000.020.040.060.080.100.12 100 50 0 50 1000.000.050.100.150.20100 50 0 50 1000.00.10.20.30.40.5 100 50 0 50 1000.010.020.030.040.050.06 100 50 0 50 1000.0250.0500.0750.1000.1250.1500.1750.200 100 50 0 50 1000.10.20.30.40.5 100 50 0 50 1000.020.040.060.080.100.12 100 50 0 50 1000.010.020.030.040.050.06 100 50 0 50 1000.00.10.20.30.4 100 50 0 50 1000.000.020.040.060.080.100.12100 50 0 50 1000.010.020.030.040.050.060.070.08 100 50 0 50 1000.000.050.100.150.200.25 100 50 0 50 1000.0100.0150.0200.0250.0300.0350.0400.045 100 50 0 50 1000.00.10.20.30.4 100 50 0 50 1000.000.020.040.060.08 100 50 0 50 1000.000.020.040.060.080.100.120.14 100 50 0 50 1000.050.100.150.200.250.30 100 50 0 50 1000.000.020.040.060.080.100.120.140.16100 50 0 50 1000.020.030.040.050.060.070.080.09 100 50 0 50 1000.000.020.040.060.080.100.12 100 50 0 50 1000.050.100.150.200.250.300.35 100 50 0 50 1000.000.050.100.150.200.25 100 50 0 50 1000.020.040.060.08 100 50 0 50 1000.000.050.100.150.200.250.30 100 50 0 50 1000.010.020.030.040.050.06 100 50 0 50 1000.0000.0250.0500.0750.1000.1250.1500.1750.200100 50 0 50 1000.0250.0500.0750.1000.1250.1500.1750.2000.225 100 50 0 50 1000.000.010.020.030.040.050.06 100 50 0 50 1000.000.010.020.030.040.050.060.07 100 50 0 50 1000.000.020.040.060.08 100 50 0 50 1000.0050.0100.0150.0200.0250.0300.0350.040 100 50 0 50 1000.000.020.040.060.080.10 100 50 0 50 1000.000.050.100.150.200.250.30 100 50 0 50 1000.000.020.040.060.080.100.120.14 (j) Land. 10000 100 50 0 50 1000.700.750.800.850.900.95 100 50 0 50 1000.00.10.20.30.4 100 50 0 50 1000.0000.0050.0100.0150.0200.0250.030 100 50 0 50 1000.00.10.20.30.40.5 100 50 0 50 1000.00050.00100.00150.00200.00250.00300.0035 100 50 0 50 1000.000.020.040.060.080.100.12 100 50 0 50 1000.0000.0250.0500.0750.1000.1250.1500.175 100 50 0 50 1000.0000.0010.0020.0030.0040.0050.0060.007100 50 0 50 1000.0000.0010.0020.0030.0040.005 100 50 0 50 1000.00000.00020.00040.00060.00080.00100.0012 100 50 0 50 1000.000000.000250.000500.000750.001000.001250.001500.00175 100 50 0 50 1000.000000250.000000500.000000750.000001000.000001250.000001500.00000175 100 50 0 50 1000.00000.00010.00020.00030.00040.00050.00060.00070.0008 100 50 0 50 1000.000.010.020.030.04 100 50 0 50 1000.00.10.20.30.40.50.60.7 100 50 0 50 1000.000.010.020.030.04100 50 0 50 1000.000000.000050.000100.000150.00020 100 50 0 50 1000.0000000.0000050.0000100.0000150.0000200.000025 100 50 0 50 1000.0000.0050.0100.0150.0200.0250.0300.035 100 50 0 50 1000.000000.000250.000500.000750.001000.001250.001500.00175 100 50 0 50 1000.000.050.100.150.200.250.30 100 50 0 50 1000.0000.0010.0020.0030.0040.005 100 50 0 50 1000.000000.000010.000020.000030.00004 100 50 0 50 1000.00000.00010.00020.00030.00040.00050.0006100 50 0 50 1000.000.010.020.030.040.050.06 100 50 0 50 1000.050.100.150.200.25 100 50 0 50 1000.000000000.000000250.000000500.000000750.000001000.000001250.000001500.000001750.00000200 100 50 0 50 1000.0000.0010.0020.0030.0040.0050.006 100 50 0 50 1000.000.010.020.030.040.050.060.070.08 100 50 0 50 1000.0000.0010.0020.0030.004 100 50 0 50 1000.00000.00250.00500.00750.01000.01250.0150 100 50 0 50 1000.000.010.020.030.04100 50 0 50 1000.000.020.040.060.080.100.120.14 100 50 0 50 1000.0000.0020.0040.0060.0080.010 100 50 0 50 1000.0000.0050.0100.0150.020 100 50 0 50 1000.000.010.020.030.040.05 100 50 0 50 1000.000000.000020.000040.000060.000080.00010 100 50 0 50 1000.000000.000020.000040.000060.000080.000100.000120.000140.00016 100 50 0 50 1000.0000010.0000020.0000030.0000040.0000050.0000060.0000070.000008 100 50 0 50 1000.000.050.100.150.200.250.300.35100 50 0 50 1000.0000000.0000050.0000100.0000150.0000200.0000250.0000300.000035 100 50 0 50 1000.920.940.960.981.00 100 50 0 50 1000.00.10.20.30.40.50.6 100 50 0 50 1000.000.020.040.060.08 100 50 0 50 1000.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.10.20.30.40.5 100 50 0 50 1000.000000.000050.000100.000150.00020100 50 0 50 1000.000000.000050.000100.000150.000200.00025 100 50 0 50 1000.00.10.20.30.40.50.6 100 50 0 50 1000.00000.00020.00040.00060.00080.00100.00120.00140.0016 100 50 0 50 1000.00000.00010.00020.00030.0004 100 50 0 50 1000.00000.00010.00020.00030.00040.0005 100 50 0 50 1000.00000.00010.00020.00030.00040.00050.00060.0007 100 50 0 50 1000.9600.9650.9700.9750.9800.9850.9900.9951.000 100 50 0 50 1000.0000.0050.0100.0150.0200.0250.030100 50 0 50 1000.00.10.20.30.40.5 100 50 0 50 1000.000000000.000000250.000000500.000000750.000001000.000001250.000001500.000001750.00000200 100 50 0 50 1000.000.010.020.030.040.05 100 50 0 50 1000.10.20.30.40.50.60.70.8 100 50 0 50 1000.000000.000050.000100.000150.000200.000250.00030 100 50 0 50 1000.0000.0020.0040.0060.0080.0100.0120.0140.016 100 50 0 50 1000.20.40.60.81.0 100 50 0 50 1000.0050.0100.0150.0200.0250.0300.035 (k) Land. 20000(l) Gen. 5000 (m) Gen. 10000 (n) Gen. 20000 Fig. 10: Result on CIFAR-10. (a) - (g) DCGAN-NS. (h) - (n)DCGAN-imba, γ = 10 .V. P REVENTING CATASTROPHIC FORGETTING Based on the reasons identified in Section III-B, we proposethe following ways to address CF problem:1) Preserve and use information from previous tasks in thecurrent task .2) Introduce prior knowledge to the game in a way suchthat old knowledge is useful for the new task and is noterased by the new task. mean/stdDCGAN 2.054/0.913DCGAN-imba, γ = 10 λ = 100 λ = 100 , γ = 10 TABLE III: Inception scores of models at iteration 50k. Theresult is averaged over 10 different runs. (a) Img. (b) Score(c) Img. (d) Score Fig. 11: Score of fixed fake images during training fromiteration 10000 to 200000. The same MLP in Fig. 2 was trainedwith SGD with learning rate e − . (a) - (b) GAN-NS. (c) -(d) GAN-0GP with λ = 100 . GAN-NS assigns random scoresto the same fake image, implying that it does not rememberinformation about this fake sample. GAN-0GP is much morestable and consistently assigns scores lower than 0.5 to oldfake samples. A. Preserving and using old information Optimizers with momentum. The update rule of SGD withmomentum g t = γ g t − + η ∇ tθ θ t +1 = θ t − g t The momentum term γ g t − is a simple form of memory thatcarries gradient information from previous training iterationsto the current iteration. When the discriminator/generator isupdated with g t , the performance of the network on previoustasks is also improved. The effectiveness of momentum inpreventing CF is demonstrated in Fig. 1h: the discriminator’sgradient pattern is more stable and similar to those of GAN-0GP and GAN-R1. Continual learning algorithms such as EWC [3] and onlineEWC [26] prevent important knowledge of previous tasks frombeing overwritten by the new task. At the end of a task T t ,online EWC computes the importance ˆ ω ti of each parameter θ ti to the task and adds a regularization term to the loss functionof task T t +1 : ω ti = α ˆ ω ti + (1 − α ) ω t − i L t +1 EW C = L t +1 + λ (cid:88) i ω ti ( θ i − θ ti ) where θ ti is the value of θ i at the end of task T t , α balancesthe importance of the current task and previous tasks, ω ti accumulates the importance of θ i throughout the trainingprocess. Because consecutive model distributions are similar,we consider a chunk of τ distributions as a task to thediscriminator. The importance ω i is computed every τ GANraining iteration. The regularizer prevents important weightsfrom deviating too far from the values that are optimal toprevious tasks while allowing less important weights to changemore freely. It helps the discriminator preserves importantinformation about old distributions. Liang et al. independentlyproposed a similar way of adapting continual learning methodsto GANs. Experiments in the paper showed that continuallearning methods improve the quality of GANs. B. Introducing prior knowledge to the game In Dirac GAN, if the discriminator has a local maximum atthe real datapoint then it can always classify the real andthe fake datapoint correctly, regardless of location of thefake datapoint. Because separating different fake distributionsfrom the target distribution requires the same knowledge,that knowledge will not be erased from the discriminator.We want to introduce to the game the knowledge that realdatapoints should be local maxima. R1 and 0GP are two waysto implement that. R1 regularizer (the third row in Table I) forces the gradientsw.r.t. a real datapoint to be , making it a local extremum ofthe discriminator. As the discriminator maximizes the score ofreal datapoints, real datapoints become local maxima of thediscriminator. Fig. 1e - 1g shows that real datapoints are alwayslocal maxima and the gradient pattern of the discriminator stayunchanged as p g moves toward p r . Fig. 5 demonstrates thesame effect of R1 on MNIST. Note that noisy images that arefar away from the real images (e.g. x + k ˆ u for k < − ) havehigher scores than real images. This is because no regularizeris applied to these noisy images. (the forth row in Table I) pushes gradientsw.r.t. datapoints on the line connecting a real datapoint x and a fake datapoint y toward . 0GP forces the score toincrease gradually as we move from y to x . During training, x is paired with different y i . Thus, the score D ( x ) is greaterthan the scores of fake datapoints in a wider neighborhood.That fixes the problem of R1 and creates wider local maxima(Fig. 4, 9). Thanh-Tung et al. [10] showed that GAN-0GPgeneralizes better than GAN-R1. Although generalization isbeyond the scope of this paper, we believe that the sharpnessof the discriminator’s landscape is related to its generalizationcapability. Prior works on generalization of neural networks[27] showed flat (wide) minima of the loss surface generalizebetter than sharp minima. Creating discriminators with widelocal maxima is a good way to improve GANs’ generalizability. WGAN-GP (the first row in Table I) uses 1-centered gradientpenalty (1GP) which pushes gradients w.r.t. datapoints on theline connecting a real datapoint x and a fake datapoint y toward , forcing the score to increase gradually from y to x . Fig. 6 shows that real datapoints are local maxima of thediscriminator. Wu et al. [28] showed that WGAN-0GP performsslightly better than WGAN-1GP. Our hypothesis is that 0GPcreates wider maxima than 1GP as it make the score on theline from y to x to change more slowly. Imbalanced weights for real and fake samples. To preventthe discriminator from forgetting distant real datapoints, wepropose to increase the weight of the loss for real datapoints: L D = γ L real + L fake (6)where γ > is an empirically chosen hyper parameter, L real , L fake are the losses for real and fake samples, re-spectively. When γ > , the discriminator is penalized moreif it assigns a low score to a real datapoint. The situationwhere real datapoints are local minima like in Fig. 10b or havelow scores like in the blue boxes in Fig. 1a - 1b will lesslikely to happen. Fig. 10k shows that the new loss successfullyhelps the discriminator to make more real datapoints localmaxima and thus improve fake samples’ quality. Table III showsthe effectiveness of imbalanced loss on CIFAR-10 dataset: itsignificantly improves Inception Score [29] and reduces thescore’s variance. The imbalanced loss is orthogonal to gradientpenalties and can be used to improve gradient penalties (thelast two rows in Table III).VI. C ONCLUSION Catastrophic forgetting is a important problem in GANs.It is directly related to mode collapse and non-convergence.Addressing catastrophic forgetting leads to better convergenceand less mode collapse. Methods such as imbalanced loss,zero centered gradient penalties, optimizers with momentum,and continual learning are effective at preventing catastrophicforgetting in GANs. 0GP helps GANs to converge to goodlocal equilibria where real datapoints are wide local maxima ofthe discriminator. The gradient penalty is a promising methodfor improving generalizability of GANs.R EFERENCES [1] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, BingXu, David Warde-Farley, Sherjil Ozair, Aaron Courville,and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems 27 ,pages 2672–2680. Curran Associates, Inc., 2014.[2] J¨urgen Schmidhuber. Learning factorial codes by pre-dictability minimization. Neural Computation , 4(6):863–879, 1992.[3] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, JoelVeness, Guillaume Desjardins, Andrei A. Rusu, KieranMilan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, DharshanKumaran, and Raia Hadsell. Overcoming catastrophicforgetting in neural networks. Proceedings of the NationalAcademy of Sciences , 114(13):3521–3526, 2017. ISSN0027-8424.[4] Michael McCloskey and Neal J. Cohen. Catastrophicinterference in connectionist networks: The sequentiallearning problem. volume 24 of Psychology of Learningand Motivation , pages 109 – 165. Academic Press, 1989.[5] Roger Ratcliff. Connectionist models of recognitionmemory: constraints imposed by learning and forgettingfunctions. Psychological review , 97(2):285, 1990.6] Robert M. French. Catastrophic forgetting in connectionistnetworks. Trends in Cognitive Sciences , 3(4):128 – 135,1999. ISSN 1364-6613.[7] Martin Arjovsky, Soumith Chintala, and L´eon Bottou.Wasserstein generative adversarial networks. In Proceed-ings of the 34th International Conference on MachineLearning , volume 70, pages 214–223. PMLR, 06–11 Aug2017.[8] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, VincentDumoulin, and Aaron C Courville. Improved training ofwasserstein gans. In Advances in Neural Information Pro-cessing Systems 30 , pages 5767–5777. Curran Associates,Inc., 2017.[9] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin.Which training methods for GANs do actually converge?In Proceedings of the 35th International Conference onMachine Learning , volume 80 of Proceedings of MachineLearning Research , pages 3478–3487, Stockholmsmssan,Stockholm Sweden, 10–15 Jul 2018. PMLR.[10] Hoang Thanh-Tung, Truyen Tran, and Svetha Venkatesh.Improving generalization and stability of generativeadversarial networks. In International Conference onLearning Representations , 2019.[11] Vaishnavh Nagarajan and J. Zico Kolter. Gradient descentgan optimization is locally stable. In Advances in NeuralInformation Processing Systems 30 , pages 5585–5595.Curran Associates, Inc., 2017.[12] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner,Bernhard Nessler, and Sepp Hochreiter. Gans trained by atwo time-scale update rule converge to a local nash equi-librium. In Advances in Neural Information ProcessingSystems 30 , pages 6626–6637. Curran Associates, Inc.,2017.[13] Lars Mescheder, Sebastian Nowozin, and Andreas Geiger.The numerics of gans. In Advances in Neural Informa-tion Processing Systems 30 , pages 1825–1835. CurranAssociates, Inc., 2017.[14] Ari Seff, Alex Beatson, Daniel Suo, and Han Liu.Continual learning in generative adversarial nets. CoRR ,abs/1705.08395, 2017.[15] Kevin J Liang, Chunyuan Li, Guoyin Wang, and LawrenceCarin. Generative Adversarial Network Training isa Continual Learning Problem. arXiv e-prints , art.arXiv:1811.11083, Nov 2018.[16] Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, andYi Zhang. Generalization and equilibrium in generativeadversarial nets (GANs). In Proceedings of the 34th In-ternational Conference on Machine Learning , volume 70,pages 224–232. PMLR, 06–11 Aug 2017.[17] Sanjeev Arora, Andrej Risteski, and Yi Zhang. Do GANslearn the distribution? some theory and empirics. In International Conference on Learning Representations ,2018.[18] Pengchuan Zhang, Qiang Liu, Dengyong Zhou, Tao Xu,and Xiaodong He. On the discrimination-generalizationtradeoff in GANs. In International Conference on Learning Representations , 2018.[19] William Fedus, Mihaela Rosca, Balaji Lakshminarayanan,Andrew M. Dai, Shakir Mohamed, and Ian Goodfellow.Many paths to equilibrium: GANs do not need to decreasea divergence at every step. In International Conferenceon Learning Representations , 2018.[20] Alec Radford, Luke Metz, and Soumith Chintala. Unsu-pervised representation learning with deep convolutionalgenerative adversarial networks. CoRR , abs/1511.06434,2015.[21] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.Deep learning face attributes in the wild. In Proceedingsof International Conference on Computer Vision (ICCV) ,December 2015.[22] Alex Krizhevsky et al. Learning multiple layers of featuresfrom tiny images. 2009.[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and JianSun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer visionand pattern recognition , pages 770–778, 2016.[24] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehti-nen. Progressive growing of GANs for improved quality,stability, and variation. In International Conference onLearning Representations , 2018.[25] Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, JoshuaSusskind, Wenda Wang, and Russell Webb. Learning fromsimulated and unsupervised images through adversarialtraining. In Proceedings of the IEEE conference oncomputer vision and pattern recognition , pages 2107–2116, 2017.[26] Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina,Agnieszka Grabska-Barwinska, Yee Whye Teh, RazvanPascanu, and Raia Hadsell. Progress & compress: A scal-able framework for continual learning. In Proceedings ofthe 35th International Conference on Machine Learning ,volume 80, pages 4528–4537. PMLR, 10–15 Jul 2018.[27] Sepp Hochreiter and J¨urgen Schmidhuber. Flat minima. Neural Computation , 9(1):1–42, 1997.[28] Jiqing Wu, Zhiwu Huang, Janine Thoma, Dinesh Acharya,and Luc Van Gool. Wasserstein divergence for gans. In Proceedings of the European Conference on ComputerVision (ECCV) , pages 653–668, 2018.[29] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, VickiCheung, Alec Radford, Xi Chen, and Xi Chen. Improvedtechniques for training gans. In Advances in NeuralInformation Processing Systems 29 , pages 2234–2242.Curran Associates, Inc., 2016. PPENDIX This section includes figures for different GANs. The generalconfiguration for all experiments are shown in Table IV. Hyperparameters specific to each experiment is shown in the captionof the corresponding figure.In each figure, the ’Real’ subfloat shows real samples fromMNIST dataset. Each cell in a ’Landscape’ subfloat shows aslice of the landscape - the value of f ( k ) , k ∈ [ − , ,for the corresponding real sample at the specified iteration.Each ’Generated’ subfloat shows the generated samples at thatiteration. .5 1.0 0.5 0.0 0.5 1.0 1.51.51.00.50.00.51.01.5 (a) Iter. 0 (b) Iter. 10 (c) Iter. 70 (d) Iter. 78 (e) Iter. 87 (f) Iter. 150 (g) Iter. 175 (h) Iter. 250 Fig. 12: Catastrophic forgetting in high capacity Dirac GAN. The discriminator is a 1 hidden layer neural network with LeakyReLU activation function and 2 hidden neurons. Although the discriminator has enough capacity to become a non-monotonicfunction, catastrophic forgetting makes it a monotonic function. High capacity Dirac GAN still oscillates around the equilibrium. Architecture 3 hidden layer MLPHidden layer activation ReLUOutput layer activation Sigmoid for GAN-NS, Linear for WGANNumber of hidden neurons 512Latent dimensionality 50Optimizer ADAM with β − . , β − . Learning rate × − Batch size 64 TABLE IV: Experiments configuration (a) Iteration 0 (b) Iteration 3000 (c) Iteration 3500 (d) Iteration 4000 (e) Iteration 4500 (f) Iteration 5000 (g) Iteration 5500 (h) Iteration 10000 (i) Iteration 20000 Fig. 13: Catastrophic forgetting on the 8 Gaussian dataset. (a) Iteration 0 (b) Iteration 500 (c) Iteration 1000 (d) Iteration 2500 (e) Iteration 5000 (f) Iteration 10000 Fig. 14: GAN-R1 with λ = 10 on 8 Gaussian dataset. (a) Iter. 0 (b) Iter. 400 (c) Iter. 600 (d) Iter. 1500 Fig. 15: Evolution sequence of GAN-NS with Adam on the 8 Gaussian dataset. The gradient pattern is much more stable thanthat of GAN-NS with SGD. Note that the gradients in the red box still point toward the real datapoint despite the fact that thefake datapoints are close. a) k = − (b) k = − (c) k = − (d) k = − (e) k = 0 (f) k = 10 (g) k = 20 (h) k = 50 (i) k = 100 Fig. 16: Real examples with different levels of noise. a) Real 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (b) Landscape 5000(c) Generated 50000 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (d) Landscape 50000(e) Generated 100000 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (f) Landscape 100000g) Generated 200000 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (h) Landscape 200000 Fig. 17: GAN-NS a) Real 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (b) Landscape 5000(c) Generated 50000 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (d) Landscape 50000(e) Generated 100000 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (f) Landscape 100000g) Generated 200000 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (h) Landscape 200000 Fig. 18: GAN-R1, λ = 100 a) Real 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (b) Landscape 5000(c) Generated 50000 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (d) Landscape 50000(e) Generated 100000 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (f) Landscape 100000g) Generated 200000 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (h) Landscape 200000 Fig. 19: GAN-0GP, λ = 100 . a) Real 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (b) Landscape 5000(c) Generated 50000 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (d) Landscape 50000(e) Generated 100000 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (f) Landscape 100000g) Generated 200000 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (h) Landscape 200000(h) Landscape 200000 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (h) Landscape 200000(h) Landscape 200000 Fig. 20: GAN-0GP with λ = 10 . a) Real 100 50 0 50 10025201510505 100 50 0 50 100151050 100 50 0 50 1002520151050510 100 50 0 50 1002520151050 100 50 0 50 10025201510505 100 50 0 50 100302520151050 100 50 0 50 100201510505 100 50 0 50 10020151050100 50 0 50 1002520151050510 100 50 0 50 10025201510505 100 50 0 50 100201510505 100 50 0 50 10025201510505 100 50 0 50 100201510505 100 50 0 50 10025201510505 100 50 0 50 10025201510505 100 50 0 50 1002520151050100 50 0 50 10025201510505 100 50 0 50 10025201510505 100 50 0 50 10025201510505 100 50 0 50 10020151050 100 50 0 50 10020151050510 100 50 0 50 1003025201510505 100 50 0 50 10020151050 100 50 0 50 10025201510505100 50 0 50 10020151050 100 50 0 50 100302520151050 100 50 0 50 100252015105051015 100 50 0 50 10020151050 100 50 0 50 10025201510505 100 50 0 50 1003025201510505 100 50 0 50 1002520151050 100 50 0 50 100201510505100 50 0 50 10025201510505 100 50 0 50 100201510505 100 50 0 50 100252015105051015 100 50 0 50 100201510505 100 50 0 50 10025201510505 100 50 0 50 100201510505 100 50 0 50 10020151050 100 50 0 50 1002520151050510100 50 0 50 100201510505 100 50 0 50 10020151050510 100 50 0 50 10020151050 100 50 0 50 10025201510505 100 50 0 50 10020151050 100 50 0 50 10020151050 100 50 0 50 100201510505 100 50 0 50 1002520151050100 50 0 50 10025201510505 100 50 0 50 1002520151050 100 50 0 50 10020151050 100 50 0 50 10020151050 100 50 0 50 10020151050 100 50 0 50 1002520151050 100 50 0 50 1002520151050 100 50 0 50 10025201510505100 50 0 50 10025201510505 100 50 0 50 100201510505 100 50 0 50 1002520151050 100 50 0 50 10025201510505 100 50 0 50 10025201510505 100 50 0 50 10025201510505 100 50 0 50 100302520151050 100 50 0 50 10025201510505 (b) Landscape 5000(c) Generated 50000 100 50 0 50 100108642024 100 50 0 50 1001210864202 100 50 0 50 10042024 100 50 0 50 10064202 100 50 0 50 10012108642024 100 50 0 50 1008642024 100 50 0 50 10010.07.55.02.50.02.55.0 100 50 0 50 100864202100 50 0 50 1001086420246 100 50 0 50 1008642024 100 50 0 50 10010.07.55.02.50.02.55.07.5 100 50 0 50 10012108642024 100 50 0 50 100864202 100 50 0 50 10086420246 100 50 0 50 10086420246 100 50 0 50 1008642024100 50 0 50 1008642024 100 50 0 50 1001086420246 100 50 0 50 10064202468 100 50 0 50 10010864202 100 50 0 50 10064202 100 50 0 50 1001086420246 100 50 0 50 10064202 100 50 0 50 100108642024100 50 0 50 1001210864202 100 50 0 50 10086420246 100 50 0 50 100642024 100 50 0 50 1008642024 100 50 0 50 10064202 100 50 0 50 10064202468 100 50 0 50 100864202 100 50 0 50 10010864202100 50 0 50 100108642024 100 50 0 50 100108642024 100 50 0 50 100543210123 100 50 0 50 100108642024 100 50 0 50 100642024 100 50 0 50 10012.510.07.55.02.50.02.55.0 100 50 0 50 1008642024 100 50 0 50 10012.510.07.55.02.50.02.55.0100 50 0 50 100420246 100 50 0 50 10043210123 100 50 0 50 100864202 100 50 0 50 1008642024 100 50 0 50 1001086420 100 50 0 50 10086420246 100 50 0 50 100108642024 100 50 0 50 1001086420100 50 0 50 100642024 100 50 0 50 100642024 100 50 0 50 1001086420 100 50 0 50 10012108642024 100 50 0 50 10012.510.07.55.02.50.02.5 100 50 0 50 10086420246 100 50 0 50 10010864202 100 50 0 50 100864202100 50 0 50 1008642024 100 50 0 50 1006420246 100 50 0 50 1001086420246 100 50 0 50 1001210864202 100 50 0 50 1008642024 100 50 0 50 1006420246 100 50 0 50 100321012345 100 50 0 50 100864202 (d) Landscape 50000(e) Generated 100000 100 50 0 50 1004202468 100 50 0 50 100420246 100 50 0 50 10021012345 100 50 0 50 10042024 100 50 0 50 10042024 100 50 0 50 10021012345 100 50 0 50 10042024 100 50 0 50 10042024100 50 0 50 100420246 100 50 0 50 100642024 100 50 0 50 100420246 100 50 0 50 10043210123 100 50 0 50 10020246 100 50 0 50 10001234567 100 50 0 50 10032101234 100 50 0 50 10032101234100 50 0 50 1006420246 100 50 0 50 100420246 100 50 0 50 100202468 100 50 0 50 100321012345 100 50 0 50 100642024 100 50 0 50 10042024 100 50 0 50 100432101234 100 50 0 50 100420246100 50 0 50 10042024 100 50 0 50 100202468 100 50 0 50 10020246 100 50 0 50 1006420246 100 50 0 50 10042024 100 50 0 50 10010123456 100 50 0 50 100642024 100 50 0 50 10043210123100 50 0 50 10020246 100 50 0 50 10042024 100 50 0 50 1002024 100 50 0 50 1006420246 100 50 0 50 100420246 100 50 0 50 100420246 100 50 0 50 100420246 100 50 0 50 100420246100 50 0 50 10001234567 100 50 0 50 100432101234 100 50 0 50 10042024 100 50 0 50 100420246 100 50 0 50 10032101234 100 50 0 50 10020246 100 50 0 50 10042024 100 50 0 50 10042024100 50 0 50 10032101234 100 50 0 50 10042024 100 50 0 50 10043210123 100 50 0 50 10042024 100 50 0 50 10042024 100 50 0 50 1006420246 100 50 0 50 10032101234 100 50 0 50 10042024100 50 0 50 10020246 100 50 0 50 1004202468 100 50 0 50 100321012345 100 50 0 50 100420246 100 50 0 50 10042024 100 50 0 50 100321012345 100 50 0 50 1004202468 100 50 0 50 100420246 (f) Landscape 100000g) Generated 200000 100 50 0 50 1000246810 100 50 0 50 10020246 100 50 0 50 10021012345 100 50 0 50 100210123456 100 50 0 50 100101234567 100 50 0 50 10021012345 100 50 0 50 10010123456 100 50 0 50 10010123456100 50 0 50 10020246 100 50 0 50 1000123456 100 50 0 50 100202468 100 50 0 50 10020246 100 50 0 50 1002101234 100 50 0 50 1000246810 100 50 0 50 10010123456 100 50 0 50 10001234567100 50 0 50 1000123456 100 50 0 50 10002468 100 50 0 50 10001234567 100 50 0 50 10010123 100 50 0 50 10010123456 100 50 0 50 10042024 100 50 0 50 1001012345 100 50 0 50 10020246100 50 0 50 1002101234 100 50 0 50 10002468 100 50 0 50 10001234567 100 50 0 50 10020246 100 50 0 50 1001012345 100 50 0 50 1000246810 100 50 0 50 1003210123 100 50 0 50 10021012345100 50 0 50 10002468 100 50 0 50 10001234567 100 50 0 50 10002468 100 50 0 50 10001234567 100 50 0 50 10020246 100 50 0 50 100202468 100 50 0 50 10001234567 100 50 0 50 10002468100 50 0 50 10001234567 100 50 0 50 100101234567 100 50 0 50 100101234567 100 50 0 50 100210123456 100 50 0 50 10020246 100 50 0 50 10020246 100 50 0 50 100210123456 100 50 0 50 10021012345100 50 0 50 100210123456 100 50 0 50 1000123456 100 50 0 50 1001012345 100 50 0 50 1001.00.50.00.51.01.52.02.53.0 100 50 0 50 1000123456 100 50 0 50 10032101234 100 50 0 50 100210123456 100 50 0 50 1000123456100 50 0 50 10020246 100 50 0 50 1000246810 100 50 0 50 100012345 100 50 0 50 1002101234 100 50 0 50 100202468 100 50 0 50 100202468 100 50 0 50 100101234567 100 50 0 50 10010123456 (h) Landscape 200000 Fig. 21: WGAN-GP with λ = 10 a) Real 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (b) Landscape 5000(c) Generated 50000 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (d) Landscape 50000(e) Generated 100000 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (f) Landscape 50000g) Generated 200000 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 100 50 0 50 1000.00.20.40.60.81.0 (h) Landscape 200000(h) Landscape 200000