Exploring Crowd Co-creation Scenarios for Sketches
EExploring Crowd Co-creation Scenarios for Sketches
Devi Parikh , and C. Lawrence Zitnick Facebook AI Research Georgia [email protected] [email protected]
Abstract
As a first step towards studying the ability of humancrowds and machines to effectively co-create, we ex-plore several human-only collaborative co-creation sce-narios. The goal in each scenario is to create a digitalsketch using a simple web interface. We find that set-tings in which multiple humans iteratively add strokesand vote on the best additions result in the sketches withhighest perceived creativity (value + novelty). Lack ofcollaboration leads to a higher variance in quality andlower novelty or surprise. Collaboration without votingleads to high novelty but low quality.
Introduction
How can one best collaborate with humans in a creative pro-cess? Insights towards this can inform what roles machinescan (or should not) play when co-creating with humans.Specifically, we consider a scenario where agents taketurns collaboratively drawing a sketch on a simple web in-terface (Figure 1). During each iteration, multiple agentspropose strokes to add to the sketch. Agents then vote onthe proposals, and the preferred set of strokes is added tothe sketch. This process is repeated for a fixed number ofiterations to create a final sketch.The roles of creating stroke proposals and voting couldeach be fulfilled by either humans (H) or machines (M). Bor-rowing terminology from Generative Adversarial Networks(Goodfellow et al. 2014), we can call the former role a gen-erator (G), and the latter a discriminator (D). This allows for4 { H,M } × {
G,D } co-creation scenarios. Further, differentindividuals could play the role of generators/discriminatorsacross iterations, leading to crowd co-creation.In this work, as a step towards human-machine co-creation, we study various human-human crowd co-creationscenarios. In the first, Individual , a single human creates theentire sketch (no discriminator D, and no crowd). Second,in
Collaborative the sketch is generated by multiple hu-man agents (crowd) iteratively taking turns adding strokes.That is, all the agents act as generators G and there is novoting or discriminator D. The third,
Collaborative + vot-ing , is where multiple human agents (generators) proposenew strokes at each iteration. Another set of human agents(discriminators) vote on which set of strokes to add to thesketch. Finally, we explore
Individual with collaborative
Figure 1: As a first step towards human-machine co-creation, we explore human-human collaboration for creat-ing digital sketches on a simple web interface shown above.Video: https://youtu.be/9fikuKPYPd0 prompts , for which the crowd is involved indirectly. A sin-gle human creates the entire sketch, but by following textprompts that describe the evolution of a sketch that was cre-ated in the
Collaborative scenario.We evaluate the qualitative difference between thesketches produced via these four scenarios. We findthat the collaborative setting with a voting mechanism(
Collaborative + voting ) leads to sketches that are rated byhuman subjects as most creative (and are preferred along avariety of other dimensions). The lack of either one of thesecomponents results in less creative sketches:
Individual sketches have decent quality (value) but low novelty, while
Collaborative sketches have high novelty but low value.
In-dividual with collaborative prompts results in high nov-elty but even worse quality. Overall, among these four sce-narios,
Collaborative + voting best hits the sweet spot forcreativity: value + novelty (Boden 1992).
Related Work
In the context of crowd-based sketching, (Tuite and Smith2012) analyze user actions and large-scale behavior patternsin 50k sketches from Sketch-a-bit, a collaborative mobiledrawing application. Different from our incremental contri- a r X i v : . [ c s . A I] M a y igure 2: Few iterations of a sketch being created in the Collaborative + voting scenario. Once a parent sketch gets fivechildren, it gets selected as the next iteration of the sketch (black outline), and the five children become the parents for the nextiteration. Temporal visualization: https://youtu.be/JQmGALAhhMU . Examples with all iterations: Figures 7 and 8.bution + voting mechanisms, (Yu and Nickerson 2011) and(Gingold et al. 2012) explore combination and averaging ofsketches respectively as collaboration strategies.Several AI systems have been trained to recognizesketches (e.g., models trained on The Quick, Draw!Dataset ). These may form useful building blocks for thenext stages of our work. However, as seen in Figure 3,our sketches tend to be complex scenes and often abstractas opposed to concrete individual objects, which has beenthe focus of most existing work in automatic sketch recog-nition. There is also work on generating images based onsketches (Chen and Hays 2018).(Davis et al. 2016) employ a cognitive science frame-work called participatory sense-making to study co-creationin sketches. Central to their study is the back and forth inter-action (dialog) between the human and machine as they taketurns. Our work is focussed on a crowd setting where no twoagents interact again in the future. (Karimi et al. 2020) studyhuman-AI co-creativity in the context of humans sketchingfor a particular design goal. Our work falls in the categoryof “casual creators” (Compton and Mateas 2015) – systemsthat support exploratory as opposed to goal-driven creativity. Sketching Interface
Human agents create sketches using the JavaScript based in-terface shown in Figure 1. Strokes can be varied across fourthicknesses and ten colors and have a paint-like texture. Thenumber and length of the strokes an agent may draw is lim-ited during each iteration. Feedback on how close they areto the cutoff is provided in real-time by the stroke limit bar.Thicker strokes count more towards the limit. Strokes drawnby the agent during the current iteration may be undone. See https://youtu.be/9fikuKPYPd0 for a video of theinterface. Our interface is publicly available.
Co-creation Scenarios
We explore four scenarios for collaborative human-humansketch co-creation. In every scenario, the sketch starts witha blank canvas. During each iteration, a limited number ofstrokes may be added. The limit roughly corresponds to five https://github.com/googlecreativelab/quickdraw-dataset medium-thickness strokes spanning the width of the canvas.30 iterations are used to create each sketch. Unless statedotherwise, we collected 20 sketches for each scenario. Allour studies were conducted on Amazon Mechanical Turk.Subjects can not submit their work till they have contributedthe required amount of strokes to the canvas. Individual.
The entire sketch is created by a single indi-vidual. That is, a single human agent adds all 30 iterations ofstrokes to “Create a beautiful, detailed, coherent painting!”.
Collaborative.
A different human agent contributesstrokes for each iteration of the sketch. That is, 30 uniqueindividuals contribute to a sketch. The first subject sees ablank canvas and adds strokes. Every subsequent subject isshown the partial sketch and asked to add to it. They cannotundo strokes from earlier contributors. The prompt is “Let’scollectively create a beautiful, detailed, coherent painting!”.Subjects are given the additional instruction to consider thekind of painting being created and the stage of the paintingwhen deciding upon which strokes to draw.
Collaborative + voting.
Each subject contributes strokesto a sketch of their choosing from a set of five starting sketchvariations. We refer to the chosen starting sketch as a parent,and the sketch created by a subject as the chosen sketch’schild. During each iteration, sketches are gathered untila parent is selected five times. Its children then replacethe current five parents and the process is repeated. Chil-dren of parents selected less than five times are discarded.See Figure 2, Figures 7 and 8 and https://youtu.be/JQmGALAhhMU for more examples.This voting strategy allows for the most promising ver-sions of a sketch to go forward. This scenario is robust tothe strokes added by any one individual. Of course, it isalso significantly more “expensive”. In the best case sce-nario where a single parent gets all 5 children and none ofthe other parents get a child, it takes 5 times the amount ofstrokes to create a sketch compared to
Collaborative . In theworst case, all 5 parents get 4 children each before a parentgets a fifth child. This would result in 21 times the numberof strokes. In practice we found this factor to be about 12.5times. Given the increased cost, we reduced the number ofiterations in this scenario to 20 (as opposed to 30). On aver-age, 250 unique individuals contribute to a single sketch.
Individual with collaborative prompts.
A single indi-igure 3: Example sketches from four co-creation scenarios along with differences identified by human subjects betweensketches from pairs of scenarios.
Collaborative + voting involves ∼ Collaborative sketches are also shown at 20 iterations. More sketches from the four scenarioscan be seen in Figures 9, 10, 11, and 12 respectively.Figure 4: Example prompts used in the
Individual with col-laborative prompts scenario.Figure 5: Evolution of example sketches in the
Collabora-tive scenario. Left: Focus of the sketch shifts from the houseto the cat in the rain outside the house. Right: Faced withseemingly incoherent strokes, subjects emphasize structurethey see in it so subsequent subjects can add to it. Moreexamples of sketches evolving are in Figures 13 and 14.vidual creates an entire sketch using instructive text promptsprovided at each iteration. The individual is instructed to fol-low the prompts when drawing. The text prompts are gener-ated by asking another individual to describe what changedin a sketch from one iteration to the next in the
Collabora-tive scenario. All text prompts for a sketch are written bya single individual. This is an interesting hybrid of havinga single creator, but being guided through prompts that de-scribe the evolution of a sketch as created by 30 unique indi-viduals. We collected three sets of text descriptions for eachof the 20
Collaborative sketches. This resulted in a total of60
Individual with collaborative prompts sketches. In ourevaluation, we consider 20 sketches (randomly picking 1 outof the set of 3). See Figure 4 for example prompts.
Evaluation
Example sketches from these scenarios are shown in Fig-ure 3 as well as in Figures 9, 10, 11, 12. Before we dis- cuss properties of the final sketch, it is worth consideringthe evolution of a sketch as it is being created.
Collabora-tive sketches evolve in several interesting ways: what seemslike the main subject of a sketch changes in a few iterations(Figure 5, left), given seemingly incoherent strokes, sub-sequent subjects try and emphasize regions that could leadto meaningful structures in the sketch for future subjects tobuild on (Figure 5, right), and subjects use the color white orother strategies to try and cover parts of the sketch they thinkare contributing negatively to it. More examples of sketchesevolving across iterations can be found in Figures 13 and 14.To assess the qualitative differences between sketchesproduced from the 4 scenarios, we created a collage of 20sketches from each scenario (at 20 iterations for
Collabo-rative and
Collaborative + voting , 30 for the rest). Weshowed pairs of collages to subjects on Amazon MechanicalTurk and asked them to describe differences that stood out.Snippets from subjects’ responses are shown in Figure 3.For a quantitative evaluation, we showed subjects pairsof sketches from two different scenarios ( i , j ). Each sub-ject picked which sketch they prefer along 12 axes. Everypair was evaluated by 5 subjects resulting in 144,000 assess-ments: 20 (sketches from scenario i ) ×
20 (sketches fromscenario j ) × ×
12 (axes) × Collaborative + voting is preferred.
Col-laborative + voting scores well for both novelty (unusual)igure 6:
Collaborative + voting sketches are consistently preferred by human subjects over sketches from other scenariosacross a variety of dimensions, and notably are rated as most creative. Notice the high variance in
Individual sketches.and quality (better, look), which we hypothesize increasesits perceived creativity.
Individual is rated well for qualitybut scores poorly on novelty. Across 11 axes,
Individual has high variance due to differences in skill + motivationof individuals creating the sketches.
Collaborative scoreswell on novelty, but worse on quality.
Individual with col-laborative prompts does poorly across all axes except forunusual, which is visually apparent in Figure 12. Of all sce-narios,
Collaborative + voting falls in the sweet spot formaximizing creativity (value + novelty).
Discussion
In what way may a machine best contribute to the collabo-rative creation of a sketch? It is often the case that humansmay not be good at generating strokes, but can tell if a sketchlooks good or not. This may suggest using machines to gen-erate candidate strokes and having humans vote on whichversions should proceed next. The machine may also con-tribute in a manner similar to the humans in our fourth sce-nario, i.e., the machine could generate textual prompts asa human draws a sketch. The prompter can have different“personalities” based on whether it is trained on sketchesgenerated from
Individual (coherent),
Collaborative (richbut chaotic) or
Collaborative + voting (rich with subtledetails and coherent). Humans and machine can generatestrokes as a team, either in co-painting scenarios as in (Ca-bannes et al. 2019), or where the machine provides somevisual guidance as in (Lee, Zitnick, and Cohen 2011) or viasuggestions for where to draw, what colors to use, etc. as ex-plored in (Oh et al. 2018). We can also train a machine to bea discriminator: given a few different stokes from a human,select which stroke should be added to the sketch next.All our sketches started with a blank canvas. We couldinstead start sketches with a prompt (subject of the sketch,adjective describing a desired property of the sketch, a pic-ture to be used as inspiration for the sketch, etc.), and havethis prompt persist across iterations (or not).It is interesting to consider ideas of ownership in the con-text of crowd co-creation. While no one individual may feela complete sense of ownership of the final piece, crowd col-laboration may lead to a sense of community and the satis-faction of contributing to a common cause. Finally, whileour motivation was human-machine co-creation, studyinghuman-human collaboration in general is, obviously, impor-tant and interesting in and of itself. Collaborative creativeendeavors may be a fertile ground for such explorations.
References [Boden 1992] Boden, M. 1992.
The Creative Mind . London:Abacus.[Cabannes et al. 2019] Cabannes, V.; Kerdreux, T.; Thiry, L.;Campana, T.; and Ferrandes, C. 2019. Dialog on a Canvaswith a Machine. In
Creativity Workshop at NeurIPS .[Chen and Hays 2018] Chen, W., and Hays, J. 2018.SketchyGAN: Towards Diverse and Realistic Sketch to Im-age Synthesis. arXiv:1801.02753 .[Compton and Mateas 2015] Compton, K., and Mateas, M.2015. Casual Creators. In
ICCC .[Davis et al. 2016] Davis, N.; Hsiao, C.-P.; Singh, K. Y.; Li,L.; and Magerko, B. 2016. Empirically Studying Participa-tory Sense-Making in Abstract Drawing with a Co-CreativeCognitive Agent. In
IUI .[Gingold et al. 2012] Gingold, Y.; Vouga, E.; Grinspun, E.;and Hirsh, H. 2012. Diamonds From the Rough: Improv-ing Drawing, Painting, and Singing via Crowdsourcing. In
HCOMP .[Goodfellow et al. 2014] Goodfellow, I. J.; Pouget-Abadie,J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville,A. C.; and Bengio, Y. 2014. Generative Adversarial Net-works.
NIPS .[Karimi et al. 2020] Karimi, P.; Rezwana, J.; Siddiqui, S.;Maher, M. L.; and Dehbozorgi, N. 2020. Creative SketchingPartner: An Analysis of Human-AI Co-creativity. In
IUI .[Lee, Zitnick, and Cohen 2011] Lee, Y. J.; Zitnick, C. L.;and Cohen, M. F. 2011. ShadowDraw: Real-time UserGuidance for Freehand Drawing.
SIGGRAPH .[Oh et al. 2018] Oh, C.; Song, J.; Choi, J.; Kim, S.; Lee, S.;and Suh, B. 2018. I Lead, You Help But Only with EnoughDetails: Understanding the User Experience of Co-Creationwith Artificial Intelligence. In
CHI .[Tuite and Smith 2012] Tuite, K., and Smith, A. 2012. Emer-gent Remix Culture in an Anonymous Collaborative ArtSystem.
AIIDE .[van der Velde et al. 2015] van der Velde, F.; Wolf, R. A.;Schmettow, M.; and Nazareth, D. S. 2015. A SemanticMap for Evaluating Creativity.
ICCC .[Yu and Nickerson 2011] Yu, L., and Nickerson, J. V. 2011.Cooks or Cobblers? Crowd Creativity through Combina-tion. In
CHI .igure 7: Iterations of a sketch being created in the
Collaborative + voting scenario. Rows correspond to iterations. Eachmeta-column (separated by black vertical lines) shows children of the same parent. When a meta-column has five children, it’scorresponding parent (outlined in black in the previous row) is selected as the next iteration of the sketch, and the five childrenbecome the next iteration of five parents. Columns have been sorted based on number of children for clarity. See Figure 2 for aclearer visualization for a few iterations. Temporal visualization: https://youtu.be/JQmGALAhhMU .igure 8: Iterations of a sketch being created in the
Collaborative + voting scenario. Rows correspond to iterations. Eachmeta-column (separated by black vertical lines) shows children of the same parent. When a meta-column has five children, it’scorresponding parent (outlined in black in the previous row) is selected as the next iteration of the sketch, and the five childrenbecome the next iteration of five parents. Columns have been sorted based on number of children for clarity. See Figure 2 for aclearer visualization for a few iterations. Temporal visualization: https://youtu.be/JQmGALAhhMU .igure 9: Example sketches from the
Individual scenario. Each sketch was created by a single individual over 30 iterations.igure 10: Example sketches from the
Collaborative scenario. Each sketch was created by 30 individuals over 30 iterations.igure 11: Example sketches from the
Collaborative + voting scenario. Each sketch was created on average by 250 individualsover 20 iterations. See text for details.igure 12: Example sketches from the
Individual with collaborative prompts scenario. Each sketch was created by 30individuals following text prompts. The text prompts described how sketches from the
Collaborative scenario changed fromone iteration to the next. See text for details.igure 13: A sketch being iteratively created in the
Collaborative scenario. Left to right, top to bottom.igure 14: A sketch being iteratively created in the