Modeling Cloud Reflectance Fields using Conditional Generative Adversarial Networks
Victor Schmidt, Mustafa Alghali, Kris Sankaran, Tianle Yuan, Yoshua Bengio
PPublished at ICLR 2020’s Climate Change AI Workshop M ODELING C LOUD R EFLECTANCE F IELDS USING C ONDITIONAL G ENERATIVE A DVERSARIAL N ET - WORKS
Victor Schmidt ∗ , Mustafa Alghali ∗ , Kris Sankaran & Yoshua Bengio Mila, Université de Montréal {schmidtv, sankarak}@mila.quebec, [email protected]
Tianle Yuan
NASA Goddard Space Flight CenterUniversity of Maryland Baltimore County [email protected] A BSTRACT
We introduce a conditional Generative Adversarial Network (cGAN) approach togenerate cloud reflectance fields (CRFs) conditioned on large scale meteorologicalvariables such as sea surface temperature and relative humidity. We show thatour trained model can generate realistic CRFs from the corresponding meteoro-logical observations, which represents a step towards a data-driven framework forstochastic cloud parameterization.
NTRODUCTION
Global Climate Models (GCMs) are one of the most important tools available to understand andanticipate the consequences of climate change, including changes in precipitation, increases intemperatures, and acceleration in glacial melting [22]. One of the key physical principles thesemodels rely on is the Earth’s energy balance [14]: in short, the difference between how much energythe earth receives and how much it emits. In this context, it is paramount to model clouds accuratelyas they both reflect energy coming to the Earth and the infrared radiations it radiates [18]. However,as physical processes at play in cloud composition and evolution typically range from − to m,direct simulation of their behavior can consume up to 20% of a GCM’s computations - dependingon their time and spatial scales [1, 4, 21]. Various efforts have tried to address this challenge. Thisincludes traditional approaches that incorporate domain knowledge to build and validate modelhypotheses using observations as well as sub-grid cloud modeling (known as super-parameterization).Alternatively, recent machine learning approaches use meteorological variables to model sub-gridclouds, thereby reducing the computational cost of super-parameterization [3, 19, 16, 25].In this paper, we extend [25], a data-driven approach to contribute to cloud modeling, focusing onone of the main features used in energy balance calculations: reflectance fields. We use ConditionalGenerative Adversarial Networks [11] to generate these reflectance fields conditioned on meteorolog-ical variables . We suggest using these generated images to extract important cloud parameters suchas optical depth. We believe our approach is a step towards building a data-driven framework that canreduce the computational complexity in traditional cloud modeling techniques.Our goal is to model reflectance fields, which in turn could be used as a proxy for cloud opticaldepth, a major component of GCMs’ energy balance computations [9, 5]. To do so, we leverage 3100aligned sample pairs X = { r i , m i } , where meteorological data m i are collocated with reflectances r i .Each m i is a × × matrix, representing 42 measurements from MERRA-2 [6] (see Table1) along with longitude and latitude to account for the Earth’s movement relative to the satellite . ∗ Equal contribution The code is available on Github: https://github.com/krisrs1128/clouds_dist As the earth rotates, the actual geographical locations on Earth change pixel position in the data. a r X i v : . [ phy s i c s . a o - ph ] A p r ublished at ICLR 2020’s Climate Change AI WorkshopOn the other hand r i , is a × × matrix representing each location’s reflectance at RGBwavelengths (680, 550 and 450 nm ) as measured by the Aqua dataset [17]. One could considerworking in a Supervised Learning setting to learn a deterministic mapping f : m i (cid:55)→ r i ; howevergiven the chaotic nature of climate, we need not a point estimate of the potential cloud distribution onearth, but rather an ensemble of likely scenarios given initial conditions. This motivates a generativeapproach using conditional GANs. ODELING REFLECTANCE FIELDS
ETWORK
Architecture : motivated by Ronneberger et al. in [20], we use a U-Net as conditional generator. TheU-Net architecture helps our generator capture global context, and skip connections allow localization.All of the convolution modules in our U-Net implementation consist of the same building blocks: a3x3 convolutional layer followed by padding - which eliminates the need for cropping - followed bybatch normalization, leaky ReLU, and a dropout layer with 0.25 probability.
Source of stochasticity : we introduce stochasticity in the generator only through the dropout layersat both training and test times, i.e we do not use noise input vectors. As observed by Isola et al. [10]dropout introduces diversity in the output of conditional GANs
Checkerboard artifacts : a direct implementation of this generator results in checkerboard artifacts,a result of the use of transposed convolutions to upsample the feature maps in the U-Net expansionpath [15]. We solve this problem by replacing transposed convolution with a resize operation of thefeature maps using 2d nearest neighbor interpolation, followed by a convolution as proposed in [15].
Discriminator : we use a multi-scale discriminator as proposed by Wang et al. in [24]. Thisintroduces 3 discriminators with identical network structure operating on different input scales:one discriminator operates on the raw input image, while the other two operate on the raw imagedownsampled by factors of 2 and 4, using average pooling with a stride of 2. The motivation behindusing discriminators at different scales is to provide the generator with better guidance both in thescale of global context and finer details in the image.2.2 T
RAINING
Training objectives
To train our generator, we use a weighted objective function composed of twolosses: a non-saturating GAN loss and a matching loss:1.
Non-saturating adversarial loss . We experimented with two types of adversarial loss: thehinge loss of [12] and the least square loss (LSGAN) of [13]. We observe better performancewith LSGAN. In figure 6, we also see that least squares loss is more stable during training.2. L matching loss . We use L loss between generated and true reflectance images, whichencourages the generator to produce outputs close to the observed images from a regressionperspective. The L loss has been found to produce less blurry outputs than L loss [10]. Optimizer . As we explored various optimization strategies and regularization methods, weobserved significant improvement both in terms of convergence and in the quality of the generatedoutput by using the Extra-Adam method proposed [7] compared to Adam and SGD, see Figure 7. ESULTS AND D ISCUSSION
ISUAL ANALYSIS
Our U-Net generator, trained against a multi-scale discriminator and optimized by Extra-Adam, isable to generate visually appealing CRFs that are difficult to distinguish from true samples. On avalidation set of ground truth images, we obtain an (cid:96) loss ∼ . . Figure 1 shows 4 different pairs ( G ( m i ) , r i ) : we can see that the model is able to pick up large-scale cloud structures as well as the See code at https://github.com/GauthierGidel/Variational-Inequality-GAN (a) (b)(c) (d)
Figure 1: 4 inferences obtained from our trained model. Generated images are on the left and thecorresponding true images on the right. We can see how composition is preserved, most large cloudfields have similar shapes but differ in the details.continents and oceans beneath them. Although not as precise as the ground-truth r i , the generatedsamples exhibit similar global composition as well as local structures.In Figure 2, we generate 3 reflectance fields from the same conditioning measurements. We noticea consistent global pattern in the three samples, with variations visible in finer details. In order toquantitatively measure diversity across generations, we fix the validation set to 5 samples that areselected manually to capture different regions of the rotating earth and generate 15 samples in total: 3for each validation sample. For each set, we compute 3 metrics: pixel-wise mean, standard deviationand inter-quartile range across samples. Figure 2 shows that the model can obtain high image qualityand proximity to the original distribution, but only the cost of low diversity.Our model still has limitations, such as blurriness and small size checkerboard artifacts. We believethe reasons for this are:1. More training samples are needed to represent such a high dimensional distribution, i.e 3100samples are not enough to train a deep U-Net generator ( ∼ . million parameters) anddiscriminator ( ∼ . million parameters).2. More hyperparameter tuning , including architectural choices of the generator and discrimina-tor to ensure the right capacity balance that lead to a long lasting game and avoid prematurelysaturated learning.3.
Further training – we can see that the discriminator loss still slightly oscillates after satura-tion points and eventually decreases with number of steps as shown in 10c.3.2 S
PECTRAL ANALYSIS
Although visual inspection techniques can give insight into GAN performance, it is an expensive,cumbersome, and subjective measure [2]. We address this issue by comparing the frequency spectrumof true and generated samples using 2D Discrete Fourier transform (DFT). This allows us to comparethe images’ geometric structures by examining the contribution of frequency components [8]. Wecompare the the magnitudes of the 2D DFT calculated from the grayscale versions of the true andgenerated images, and compare the histograms of the calculated magnitudes, their means, variancesand the logarithmic average L distances. In figure 4 we observe that our generated images haveconsistent and similar DFT distributions to those of their corresponding true images, with a verysmall average L distance. 3ublished at ICLR 2020’s Climate Change AI WorkshopFigure 2: Reflectance fields generated by conditioning on the same input (noise comes from dropout,which is kept at test time). ━ val_sample_dist_iqd ━ val_sample_dist_mean ━ val_sample_dist_std
10k 20k 30k 40k Step00.050.10.150.2
Figure 3: Plot of the inter-quartile distance, mean, and standard deviation of 15 generated samples atdifferent steps during the model’s training.Figure 4: Comparison of DFTs (from left to right: image, frequency magnitudes, histogram ofmagnitudes), with real data in the top row and a generated reflectance field (from the real data’sassociated measurements)in the bottom one
ONCLUSION AND FUTURE WORK
We show that using conditional GANs to model CRFs can be an effective approach towards buildinga data-driven framework. We think our approach could significantly help improve the computationtime of clouds modeling in global climate models.Future work includes increasing the size of the dataset and exploitation of the temporal structurein our data in two ways: by adding date and time as extra labels to the input variable, and byusing temporal cross validation [23] to validate our generator’s ability to predict possible changes incloud distribution over time. We also plan to increase the diversity in the generated ensembles byincorporating input noise channels as an extra source of stochasticity. To address what we suspect tobe mode collapsing in our network (the matching loss discourages the exploration of other potentialmodes in the data) we suggest using staged training where an adaptive weight for the matching lossencourages the generator to regress onto true images during early stages of the training, eventuallydecreasing to zero as training progresses. 4ublished at ICLR 2020’s Climate Change AI Workshop R EFERENCES [1] Akio Arakawa. The cumulus parameterization problem: Past, present, and future.
Journal ofClimate , 17(13):2493–2525, 2004.[2] Ali Borji. Pros and cons of gan evaluation measures.
Computer Vision and Image Understanding ,179:41–65, 2019.[3] Noah D Brenowitz and Christopher S Bretherton. Prognostic validation of a neural networkunified physics parameterization.
Geophysical Research Letters , 45(12):6289–6298, 2018.[4] Christopher S Bretherton. Insights into low-latitude cloud feedbacks from high-resolutionmodels.
Philosophical Transactions of the Royal Society A: Mathematical, Physical andEngineering Sciences , 373(2054):20140415, 2015.[5] T Corti, BP Luo, Th Peter, H Vömel, and Q Fu. Mean radiative energy balance and verticalmass fluxes in the equatorial upper troposphere and lower stratosphere.
Geophysical researchletters , 32(6), 2005.[6] Ronald Gelaro, Will McCarty, Max J Suárez, Ricardo Todling, Andrea Molod, LawrenceTakacs, Cynthia A Randles, Anton Darmenov, Michael G Bosilovich, Rolf Reichle, et al. Themodern-era retrospective analysis for research and applications, version 2 (merra-2).
Journal ofClimate , 30(14):5419–5454, 2017.[7] Gauthier Gidel, Hugo Berard, Gaëtan Vignoud, Pascal Vincent, and Simon Lacoste-Julien.A variational inequality perspective on generative adversarial networks. arXiv preprintarXiv:1802.10551 , 2018.[8] CR Gonzalez and ER Woods. Digital image processing, addison-wesley publishing company.1991.[9] Dennis L Hartmann, Maureen E Ockert-Bell, and Marc L Michelsen. The effect of cloud typeon earth’s energy balance: Global analysis.
Journal of Climate , 5(11):1281–1304, 1992.[10] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation withconditional adversarial networks. In
Proceedings of the IEEE conference on computer visionand pattern recognition , pp. 1125–1134, 2017.[11] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation withconditional adversarial networks. In
Proceedings of the IEEE conference on computer visionand pattern recognition , pp. 1125–1134, 2017.[12] Jae Hyun Lim and Jong Chul Ye. Geometric gan. arXiv preprint arXiv:1705.02894 , 2017.[13] Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley.Least squares generative adversarial networks. In
Proceedings of the IEEE InternationalConference on Computer Vision , pp. 2794–2802, 2017.[14] Gerald R North, Robert F Cahalan, and James A Coakley Jr. Energy balance climate models.
Reviews of Geophysics , 19(1):91–121, 1981.[15] Augustus Odena, Vincent Dumoulin, and Chris Olah. Deconvolution and checkerboard artifacts.
Distill , 1(10):e3, 2016.[16] Paul A O’Gorman and John G Dwyer. Using machine learning to parameterize moist convection:Potential for modeling of climate, climate change, and extreme events.
Journal of Advances inModeling Earth Systems , 10(10):2548–2563, 2018.[17] S Platnick, KG Meyer, MD King, G Wind, N Amarasinghe, B Marchant, GT Arnold, Z Zhang,PA Hubanks, RE Holz, et al. The modis cloud optical and microphysical products: Collection 6updates and examples from terra and aqua, ieee t. geosci. remote, 55, 502–525, 2017.[18] VLRD Ramanathan, RD Cess, EF Harrison, P Minnis, BR Barkstrom, E Ahmad, and D Hart-mann. Cloud-radiative forcing and climate: Results from the earth radiation budget experiment.
Science , 243(4887):57–63, 1989. 5ublished at ICLR 2020’s Climate Change AI Workshop[19] Stephan Rasp, Michael S Pritchard, and Pierre Gentine. Deep learning to represent subgridprocesses in climate models.
Proceedings of the National Academy of Sciences , 115(39):9684–9689, 2018.[20] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks forbiomedical image segmentation. In
International Conference on Medical image computing andcomputer-assisted intervention , pp. 234–241. Springer, 2015.[21] Tapio Schneider, João Teixeira, Christopher S Bretherton, Florent Brient, Kyle G Pressel,Christoph Schär, and A Pier Siebesma. Climate goals and computing the future of clouds.
Nature Climate Change , 7(1):3–5, 2017.[22] Thomas F Stocker, Dahe Qin, Gian-Kasper Plattner, Melinda Tignor, Simon K Allen, JudithBoschung, Alexander Nauels, Yu Xia, Vincent Bex, Pauline M Midgley, et al. Climate change2013: The physical science basis.
Contribution of working group I to the fifth assessment reportof the intergovernmental panel on climate change , 1535, 2013.[23] Sudhir Varma and Richard Simon. Bias in error estimation when using cross-validation formodel selection.
BMC bioinformatics , 7(1):91, 2006.[24] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.High-resolution image synthesis and semantic manipulation with conditional gans. In
Pro-ceedings of the IEEE conference on computer vision and pattern recognition , pp. 8798–8807,2018.[25] T. Yuan, H. Song, D. Hall, V. Schmidt, K. Sankaran, and Y. Bengio. Artificial intelligencebased cloud distributor (ai-cd): probing clouds with generative adversarial networks.
AGU FallMeeting 2019 , 2019. 6ublished at ICLR 2020’s Climate Change AI Workshop
A D
ATA PROCESSING
During data processing, we Winsorize reflectance data to remove artifacts that are present in sometraining samples due to sensor noise, clipping values to the 95th percentile for each channel. This isnecessary as meteorological variables have different scales, see Figure 5. We standardize channels tohave values in [ − , and zero mean. In order to to avoid introducing unnecessary bias from valuesoutside the earth disk, we first crop the images to cut off most of these values and then upsamplethem again to their original size using 2D nearest neighbors, replacing the remaining values with − (mean - 3x standard deviation) to avoid introducing any unnecessary bias in the data distribution.We used running statistics to compute the summary statistics of the data due to the huge size of inputtensors, which do not fit in 16GB GPU memory at one time. We also increased the number of dataloader workers to 12; this accelerates the data loading process by × .Table 1: Description of input componentsName Description Number of channelsU, V Wind components in 10 atmospheric levels 20T Temperature in 10 atmospheric levels 10RH Relative-humidity in 10 atmospheric levels 10SA Scattering angle 1TS Surface Temperature 1Lat, Long Latitude and Longitude 2Figure 5: Histograms of six input variables shows the variance in scales.7ublished at ICLR 2020’s Climate Change AI Workshop B H
YPER - PARAMETER COMPARISONS (a) L1 matching loss (b) Total weighted generator loss.
Figure 6: Comparison between the hinge loss (green) and the least squares loss (purple) on modeltraining stability and convergence, we observe that the latter performs better both in optimization ofthe L loss and the total weighted generator loss. We configured Adam and ExtraAdam to use β =0.5, and β = 0.99 in all experiments. (a) L matching loss. ━ adam ━ extraadam ━ extrasgd
10k 20k 30k 40k Step-101234 (b) Generator adversarial loss. ━ adam ━ extraadam ━ extrasgd
10k 20k 30k 40k Step00.511.52 (c) Discriminator loss.(d) Adam (e) ExtraSGD (f) ExtraAdam (g) Real earth
Figure 7: The losses of 3 experiments with 3 different optimizers: Adam, ExtraSGD, and ExtraAdam,along with generated outputs for each experiment conditioned on a fixed meteorological input.ExtraAdam shows better convergence, less oscillating losses, and more visually appealing outputrelative to Adam and ExtraSGD.
Regression vs. hallucinated features
The λ /λ ratio in the generator’s weighted objectivefunction plays an important role in the behavior of our generator; experiments with large ratios in therange of [1 , behave like supervised models where we regress the generated images with L loss,while small ratios of ≤ . tend to give the generator more freedom to explore the distribution ofinterest, without being penalized for not matching low frequency details. This causes the generatorto hallucinate features that do not exist in the true images (Figure 8). This behavior matches ourexpectations: 3100 samples is not sufficient to learn the conditional distribution of such variable andhigh-dimensional data. Architecture Summary
The main components of our architecture are summarized in Figure 9.
Sharpness of generated images
Generating sharp images that can show complicated and detailedclouds structures such as spinning clouds is both important and challenging. We address this challengeby carefully choosing the discriminator learning rate to avoid saddle point convergence and non-convergence, which result in bad generations. We hypothesize that the generator might have not beentrained long enough to learn such micro-level details and thus generate blurry output. In most of these8ublished at ICLR 2020’s Climate Change AI Workshop ━ ━
2k 4k 6k 8k 10kStep00.20.40.60.811.21.4 (a) L matching loss (b) λ /λ = 5 (c) λ /λ = 0 . (d) real Figure 8: Two experiments with two different λ /λ ratios shows stability of the discriminator lossfor larger ratios – resulting in more realistic images, see (b) –, while small ratios produce hallucinatedand unrealistic images, see (c).Figure 9: The overall architecture includes two models, a U-Net generator and multiscale discrimina-tor, and the optimization objective combines a cGAN loss with an L matching loss.cases, we observe the generator is the dominant player, and the discriminator is fooled at saturationpoints, see the example shown in Figure 10.
10k 20k 30k 40k Step00.10.20.30.40.5 (a) L matching loss
10k 20k 30k 40k Step00.511.522.5 (b) Generator loss
10k 20k 30k 40k Step-2-1012 (c) Discriminator loss (d) fake (e) real
Figure 10: An example experiment with a large discriminator learning rate (0.001) shows earlysaturation of the discriminator loss after ∼∼