Neuro-Symbolic Generative Art: A Preliminary Study
NNeuro-Symbolic Generative Art: A Preliminary Study
Gunjan Aggarwal and Devi Parikh [email protected] [email protected] Adobe Georgia Tech and Facebook AI Research
Abstract
There are two classes of generative art approaches: neu-ral, where a deep model is trained to generate samplesfrom a data distribution, and “symbolic” or algorith-mic, where an artist designs the primary parameters andan autonomous system generates samples within theseconstraints. In this work, we propose a new hybridgenre: neuro-symbolic generative art. As a prelimi-nary study, we train a generative deep neural networkon samples from the symbolic approach. We demon-strate through human studies that subjects find the fi-nal artifacts and the creation process using our neuro-symbolic approach to be more creative than the sym-bolic approach 61% and 82% of the time respectively.
Introduction
Generative art refers to art generated using code, and typi-cally includes an element of chance. Interactive versions ofthese systems can be viewed as Casual Creators (Comptonand Mateas 2015). There are two dominant approaches forgenerative visual art. The first uses deep neural networksto generate images from a distribution that mimics train-ing data. Indeed, generative artists train models on specificphotographs they take or collect (e.g., Helena Sarin, Rob-bie Barrat) or perturb the weights of models to create artis-tic “glitches” in the generated art (e.g., Mario Klingemann).Another source of control is the random noise input to themodel. Interpolations of two noise vectors smoothly con-trols the generation in a local neighborhood. In the secondapproach an artist defines an algorithm to generate art. Anautonomous system generates random samples using this al-gorithm. Early algorithmic artists include Georg Nees andVera Molnar. Algorithmic art is often abstract, with geo-metric structures, or repeating or recursive patterns. These“symbolic” approaches typically have explicit parameters tocontrol the generated art.To the best of our knowledge, these two approaches togenerative visual art – neural and symbolic – have beenlargely distinct. This work is a preliminary study in ex-ploring their intersection: neuro-symbolic generative art.Specifically, we train a Generative Adversarial Network(GAN) on samples generated using a symbolic approach.We hypothesize that the organic, unpredictable aesthetic as-sociated with neural approaches complements the crisp, de- Figure 1: Example neuro-symbolic generated art samples.signed aesthetic of symbolic approaches. Moreover, com-patible with data-hungry deep models, symbolic approachessupport generation of large amounts of training samples. Ex-ample generated art samples from our approach are shownin Figure 1. Our human studies show that subjects find theartifacts and the interactive creation process using the neuro-symbolic approach to be more creative 61% and 82% of thetime respectively compared to the symbolic approach.
Related Work
Neural generative models.
These include Generative Ad-versarial Networks (GANs) (Goodfellow et al. 2014), Au-toregressive Models (Salimans et al. 2017), Latent VariableModels (Kingma and Welling 2014), etc. Recent progressin GANs enables realistic natural (Brock, Donahue, andSimonyan 2019) and high resolution human face (Karras,Laine, and Aila 2019) image generation. We limit our studyto GANs. GANs to generate video game levels (Giacomello,Lanzi, and Loiacono 2018) are particularly relevant as neu- a r X i v : . [ c s . A I] J u l igure 2: Symbolic generated art. Also see Figure 7.ral models trained on procedurally generated content. Interactive GANs.
Unlike symbolic algorithmic art, GANsdo not have interpretable parameters to control the generatedart. The input latent vectors have been shown to frequentlycontain interpretable variations (Radford, Metz, and Chin-tala 2016). (H¨ark¨onen et al. 2020) showed that PCA direc-tions of style latent vectors in StyleGAN (Karras, Laine, andAila 2019) contain intuitive interpolation directions such asrotation. (Shen et al. 2019) alter facial attributes such as age,pose by editing the latent space of GANs.
Neuro-symbolic AI.
There is currently much debate aboutthe role symbolic reasoning plays in modern AI systems.Neuro-symbolic approaches are considered by some to be atthe forefront of the next wave of AI advances. Generativeart may serve as a testbed for some of these ideas.
Dataset
In this preliminary study, we use Circle Packing as our sym-bolic generator. Non-overlapping circles are placed at ran-dom locations in the image. The sizes and number of circlesof each size is specified by the artist. Circles are placed indecreasing order of size. The color of each circle is sampledfrom a palette. A user can control the art generated via thecolor palettes (5 options, containing 5 colors each), numberof colors sampled from the palette (1 to 5), and the randomseed. For each of the 5 × Approach
We experimented with different GAN model architecturesand found Progressive GAN (Karras et al. 2018) to workbest. The final generated image is 512 × ×
4, and is doubled every 37k iterations. The in-put noise is d . The learning rates for the generator anddiscriminator were 0.001 for all image resolutions. We usedifferent batch sizes for different image resolutions duringtraining: 128 till 16 ×
16 and then halved after each sub-sequent increase in resolution. The training is run for 600kiterations with Adam optimizer. We refer to the samplesgenerated from this model as Neuro-Symbolic Generations(NSG). Some examples of NSG can be found in Figure 8.Different NSG samples can be generated from the model byfeeding it different input noise vectors. A user can control the samples via the input noise vec-tor, and interpolations of two noise vectors. For interpolatedgeneration, we sample two noise vectors and then create ar-bitrary linear interpolations between the two vectors. Neuro-Symbolic Interpolations (NSI) are then generated by feedingthe model each of the interpolated latent vectors. Specifi-cally, NSI x is generated from the generative model G as x = G ( z ); z = z + α ∗ ( z − z ) z ∼ N (0 , z ∼ N (0 , α ∈ (0 , set by the user.Figures 3 and 9 show example interpolated samples. Human Evaluation
We perform evaluation of both the artifact and the user-driven interactive creation process via human studies onAmazon Mechanical Turk (AMT). Subjects were from theUS, with an AMT approval rating of 95% or higher, andhaving completed at least 5000 tasks on AMT in the past.They were paid above federal minimum wage.
Artifact Evaluation
We compare three artifacts – Sym-bolic, Neuro-Symbolic Generation (NSG), and Neuro-Symbolic Interpolated generation (NSI). These replicate thedifferent artifacts a user might create when using the sym-bolic or neuro-symbolic interactive generative art tools. Hu-man subjects were shown a pair of art pieces, one each fromrandom two of the three types. They were asked whichpiece 1) seems more different from art you’ve seen in yourlife? 2) looks better? 3) is more creative? 4) is more artis-tic? 5) seems more likely to be hand-made? Subjects werealso asked to optionally state why they felt one art was morelikely to be hand-made than the other. The study consistedof 60 pairs, equally distributed across the three pairs of ap-proaches. The study was completed by 20 unique subjects,resulting in a total of 1200 pairwise assessments.Note that if we paired arbitrary pieces from two ap-proaches, more than just the style of the art would likelydiffer (e.g., the color palette). To control this, the pairs wereformed by finding nearest neighbors across approaches us-ing color histograms. This helps minimize unrelated vari-ations, and helps focus the study on the different styles ofsamples. Example image pairs shown to subjects can befound in Figure 4. Specifically, we first generate 10k NSGsamples. We then pick a pair of samples and compute anNSI sample associated with the pair using α = 0.5. We gen-erate 10k such NSI samples. Recall that we already have10k Symbolic samples in our dataset. Now to form a NSGvs. Symbolic pair, we pick either a random NSG or Sym-bolic sample from our dataset, and find the nearest neighborfrom the pool of 10k images of the other category. Same forNSI vs. Symbolic, and NSI vs. NSG. The two images in apair are randomly shuffled before showing it to subjects.As quality control, we additionally asked subjects thenumber of colors in one of the artifacts in a pair. The numberof colors in the symbolically generated art is known, givingus a way to identify subjects not doing the task well. Beyond1 through 5, we gave subjects an added option of “Shadedcolors, so not meaningful to count.”igure 3: Neuro-symbolic interpolations (NSIs) between two neuro-symbolic generations (NSGs). Also see Figure 9.Figure 4: Example pairs shown to subjects for evaluation.The proportion of times users preferred one art form overanother is shown in Figure 5. A one-sample proportion hy-pothesis test suggests that for our sample size, a “win ratio”over 0.54 (or below 0.46) is statistically significant at 95%confidence. These are shown as a horizontal lines in thefigure. Novelty, unusualness:
We find that the human eval-uators rate NSG and NSI as being more “different” from artthey’ve seen before than the Symbolic art about 66% of thetime.
Better quality, value:
Subjects like NSG, NSI andSymbolic almost equally.
Creativity:
The third and fourthdimensions (“creative” and “artistic”) focus directly on thecreative aspect of the artifacts. We see that human sub-jects find NSG and NSI to be more creative than Symbolicart about 61% and more artistic about 63% of the times.Note that NS(G/I) and Symbolic rated similarly for qual-ity, but NS(G/I) were rated higher for novelty. We hypothe-size that this results in NS(G/I) being rated higher in creativ-ity overall (novelty + value, (Boden 2004)).
Naturalness,hand-made:
Subjects find NS(G/I) art to be more naturalor more likely to be hand-made. Based on the commentsshared, while certain subjects preferred Symbolic art as be-ing more hand-made because of “perfect coloring” , about59% of them chose the NS(G/I) art to be more likely to behand-made because it “Looks like human error with paintdripping on to another color”, “The other piece of art hassolid colors, where as the one I picked has various shades inspots.”, “mixture of color together”, “smudge” . Finally, wesee that NSI is preferred over NSG for novelty and creativity.
Different Better Creative Artistic Hand-made W i n r a t i o Neuro-Symbolic Generation (NSG) over SymbolicNeuro-Symbolic Interpolation (NSI) over SymbolicNSI over NSG
Figure 5: Artifact evaluation along 5 axes. Dashed linesdenote the band within which a win ratio is not statisticallysignificantly different than 0.5 with 95% confidence.
Creation Process Evaluation
Next, we evaluate live in-teractive generative art tools based on symbolic and our pro-posed neuro-symbolic approaches. Recall that the symbolicapproach has 2 controllable parameters: color palette andthe number of colors (maximum 5), as well as an option togenerate a new variant of the art with the same parameters bychanging the random seed. The neuro-symbolic tool has onecontrollable parameter α which generates an NSI between 2NSG pieces, and an option to sample a new pair of NSGart by sampling new noise vectors. Human subjects weregiven links to both tools (Symbolic: http://genart.cloudcv.org/symbolic , Neuro-symbolic: http://genart.cloudcv.org ) and for each, were asked to:“Find an art piece that you like a lot and share it with us!”and describe “what characteristics of your favorite art madeit stand out from others?” Additionally, subjects were askedwhich tool 1) generates better looking art? 2) generates moresurprising / unusual / unpredictable art? 3) generates morecreative art? 4) is more satisfying to work with? 50 uniquesubjects participated. Half were given the symbolic toolfirst, and the other half the neuro-symbolic tool. Both toolshad an option to add up to 5 pieces to their “favorite” galleryso users can keep track of pieces they like as they encounterthem. Users could delete pieces from the gallery to replacethem with others. They were provided an easy way to copythe URL of their favorite piece and submit it.The proportion of times subjects preferred the neuro- etter Unusual Creative Satisfying0.250.380.500.620.750.85 W i n r a t i o Neuro-Symbolic over Symbolic
Figure 6: Evaluation of the interactive creation processalong 4 axes. Win ratios outside the dashed band are sta-tistically significantly different than 0.5 at 95% confidence.symbolic (NS) tool over symbolic (S) is shown in Figure 6.Trends are similar to artifact evaluation.
Better quality,value:
They like art generated by both tools equally.
Nov-elty, unusualness:
Subjects rate NS to be more surprisingand unusual than S 68% of the time.
Creativity:
Subjectsfind the NS tool to generate more creative art than S 82% ofthe time.
Satisfying:
Interestingly, while less creative, sub-jects find S to be more satisfying to work with (albeit, notwith statistical significance). An indicative comment froma subject: “I liked the task. I found that in [NS] the colorsfelt like they mixed together more. I found that the art in [S]was more clean looking and that made it more satisfying towork with in my opinion.”
Using S is perhaps more analo-gous to “zen” (relaxing) activities, while NS may be closerto cognitively taxing creative activities. Exploring this isfuture work. Other comments about the two tools: “I hada bit more creative control with [S], while [NS] did gener-ate more interesting combinations, it was just harder to getthere predictably.”, “[NS] provided more creativity, versustaking out colors like in [S].”, “I liked in [S] the ability tochoose the number of colors. I felt that in [NS] it was alittle harder using my mouse to get the form and shape ofthe circles I wanted.”, “This was a very interesting experi-ment, especially [NS]. I kind of felt like I didn’t know whatto expect when I was trying to make my hybrid art.”
Example generated art samples beyond those shownin Figures, screen captures of our interactive generativeart tools, and “favorite” pieces created by subjects alongwith a description of why they like the pieces can befound here: https://sites.google.com/view/neuro-symbolic-art-gen . Conclusion
We present a preliminary study on neuro-symbolic gener-ative art. It combines what have typically been two dis-tinct approaches to generative visual art: neural and algo-rithmic/symbolic. We trained GANs on data generated via asymbolic approach. We evaluate the generated art and buildlive interactive generative art tools using both approaches.Human studies show that subjects find the neuro-symbolicgenerated art and creation process to be more creative than symbolic counterparts 61% and 82% of the time respec-tively. Overall, we see promising indications that neuro-symbolic generative art may be a viable new genre.
Future Work.
We will explore other symbolic art styles,and train a model over multiple styles to potentially discoverentirely novel styles. We further plan to interpolate betweentwo symbolic images instead of two neuro-symbolic images.For this, we will explore techniques that map real images tolatent representations. A user can then first design the twoends points (symbolically), and then generate an intermedi-ate piece (neurally). Neither symbolic nor neuro-symbolicapproaches alone allow for this level of control. Finally,training GANs directly on symbolic representations is aninteresting and open research question.
Acknowledgment.
Abhishek Sinha for helpful discussions.
References [Boden 2004] Boden, M. A. 2004.
The Creative Mind:Myths And Mechanisms . Psychology Press.[Brock, Donahue, and Simonyan 2019] Brock, A.; Donahue,J.; and Simonyan, K. 2019. Large Scale GAN Training ForHigh Fidelity Natural Image Synthesis. In
ICLR 2019, .[Compton and Mateas 2015] Compton, K., and Mateas, M.2015. Casual Creators. In
Proceedings of the Sixth Interna-tional Conference on Computational Creativity June , 228.[Giacomello, Lanzi, and Loiacono 2018] Giacomello, E.;Lanzi, P. L.; and Loiacono, D. 2018. DOOM LevelGeneration Using Generative Adversarial Networks. In
GEM .[Goodfellow et al. 2014] Goodfellow, I.; Pouget-Abadie, J.;Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville,A.; and Bengio, Y. 2014. Generative Adversarial Nets. In
NeurIPS .[H¨ark¨onen et al. 2020] H¨ark¨onen, E.; Hertzmann, A.; Lehti-nen, J.; and Paris, S. 2020. GANSpace: Discovering Inter-pretable GAN Controls. arXiv:2004.02546 .[Karras et al. 2018] Karras, T.; Aila, T.; Laine, S.; and Lehti-nen, J. 2018. Progressive Growing Of GANs For ImprovedQuality, Stability, And Variation. In
ICLR 2018, .[Karras, Laine, and Aila 2019] Karras, T.; Laine, S.; andAila, T. 2019. A Style-Based Generator Architecture ForGenerative Adversarial Networks. In
CVPR .[Kingma and Welling 2014] Kingma, D. P., and Welling, M.2014. Auto-Encoding Variational Bayes. In Bengio, Y., andLeCun, Y., eds.,
In ICLR 2014, .[Radford, Metz, and Chintala 2016] Radford, A.; Metz, L.;and Chintala, S. 2016. Unsupervised Representation Learn-ing With Deep Convolutional Generative Adversarial Net-works. In Bengio, Y., and LeCun, Y., eds.,
In ICLR 2016, .[Salimans et al. 2017] Salimans, T.; Karpathy, A.; Chen, X.;and Kingma, D. P. 2017. PixelCNN++: Improving The Pix-elCNN With Discretized Logistic Mixture Likelihood AndOther Modifications. In
In ICLR 2017, .[Shen et al. 2019] Shen, Y.; Gu, J.; Tang, X.; and Zhou, B.2019. Interpreting The Latent Space Of Gans For SemanticFace Editing. arXiv:1907.10786arXiv:1907.10786