CContinuous Curve Textures
PEIHAN TU,
University of Maryland, College Park
LI-YI WEI,
Adobe Research
KOJI YATANI and TAKEO IGARASHI,
University of Tokyo
MATTHIAS ZWICKER,
University of Maryland, College Park (a) Output of (b) (b) String(c) Strip (d) Output of (c) (e) Output of (f) (f) Maze(g) Branches (h) Output of (g)Fig. 1.
Example inputs and outputs of our method.
Given a small input exemplar, our algorithm can synthesize various types of continuous curve textures. Ourresults are shown in vector format. Please zoom in to see the details.
Repetitive patterns are ubiquitous in natural and human-made objects, andcan be created with a variety of tools and methods. Manual authoring pro-vides unmatched degree of freedom and control, but can require significantartistic expertise and manual labor. Computational methods can automateparts of the manual creation process, but are mainly tailored for discrete pix-els or elements instead of more general continuous structures. We proposean example-based method to synthesize continuous curve patterns fromexemplars. Our main idea is to extend prior sample-based discrete elementsynthesis methods to consider not only sample positions (geometry) but alsotheir connections (topology). Since continuous structures can exhibit highercomplexity than discrete elements, we also propose robust, hierarchicalsynthesis to enhance output quality. Our algorithm can generate a variety ofcontinuous curve patterns fully automatically. For further quality improve-ment and customization, we also present an autocomplete user interface tofacilitate interactive creation and iterative editing. We evaluate our methodsand interface via different patterns, ablation studies, and comparisons withalternative methods.CCS Concepts: •
Human-centered computing → Interactive systemsand tools ; •
Computing methodologies → Texturing .Additional Key Words and Phrases: continuous, curve, texture, pattern, syn-thesis, interface
ACM Reference Format:
Peihan Tu, Li-Yi Wei, Koji Yatani, Takeo Igarashi, and Matthias Zwicker.2020. Continuous Curve Textures.
ACM Trans. Graph.
39, 6, Article 168(December 2020), 16 pages. https://doi.org/10.1145/3414685.3417780
Repetitive patterns are fundamental for a variety of tasks in de-sign [Kazi et al. 2012; Lu et al. 2014] and engineering [Chen et al.2016; Martínez et al. 2015; Schumacher et al. 2016; Zehnder et al.2016; Zhou et al. 2014]. Manually creating these patterns provideshigh degrees of individual freedom, but can also require signifi-cant technical/artistic expertise and manual labor. These usabilitybarriers can be reduced by automatic methods that can synthesizepatterns similar to user-supplied exemplars [Barla et al. 2006; Hsuet al. 2018, 2020; Hurtut et al. 2009; Ijiri et al. 2008; Kazi et al. 2012;Lu et al. 2014; Ma et al. 2013, 2011; Suzuki et al. 2017]. However,existing techniques mainly focus on discrete patterns consistingof image pixels or shape elements, and might not apply to generalpatterns consisting of continuous curves, which can be connectedor intersected with one another.We propose an example-based method that can automaticallysynthesize continuous curve patterns from user-supplied exemplars.Similar to prior pixel/sample-based methods [Landes et al. 2013;Lu et al. 2014, 2012; Ma et al. 2013, 2011; Roveri et al. 2015; Weiet al. 2009], users can provide exemplars and have the algorithmautomatically produce results in desired sizes and shapes. However,different from previous methods and systems that are restrictedto discrete pixels/elements or limited continuous structures, ourmethod can handle both discrete elements and continuous curvesin a variety of patterns (Figure 1).
Authors’ addresses: Peihan Tu, University of Maryland, College Park; Li-Yi Wei, AdobeResearch; Koji Yatani; Takeo Igarashi, University of Tokyo; Matthias Zwicker, Universityof Maryland, College Park.© 2020 Copyright held by the owner/author(s). Publication rights licensed to ACM.This is the author’s version of the work. It is posted here for your personal use. Not forredistribution. The definitive Version of Record was published in
ACM Transactions onGraphics , https://doi.org/10.1145/3414685.3417780.ACM Trans. Graph., Vol. 39, No. 6, Article 168. Publication date: December 2020. a r X i v : . [ c s . G R ] D ec (a) Exemplar (b) Samples of (a) (c) [Ma et al. 2011] (d) [Roveri et al. 2015] (e) [Tu et al. 2019] (f) Our output samples (g) Reconstruction of (f)Fig. 2. Comparison with prior point/sample synthesis algorithms.
The methods in [Ma et al. 2011; Roveri et al. 2015; Tu et al. 2019] generate samples withoutconsidering their connections in the exemplars (b). Thus, their results, as shown in (c), (d), and (e), preserve the sample distributions less well than ours in (f).It is also unclear how to reconstruct continuous curve patterns from (c), (d), and (e). (g) shows the curve reconstruction from (f).
Our main idea is to extend prior sample-based element synthesismethods [Hsu et al. 2018; Kazi et al. 2012; Ma et al. 2013, 2011] toconsider not only sample positions (geometry) but also their con-nections (topology) in all major algorithm components, includingpattern representation, neighborhood similarity, and synthesis opti-mization consisting of search and assignment steps. Our algorithmuses a graph representation for both topology synthesis and geomet-ric path reconstruction for general continuous patterns, in contrastto the graph representations in [Hsu et al. 2018, 2020] that onlyapply to discrete elements. Since continuous patterns can exhibithigher complexity than discrete elements, we propose robust, hier-archical synthesis [Wei and Levoy 2000, 2001] to enhance outputquality.Automatically generated outputs, although convenient, might nothave sufficient quality or fit what users have in mind for their par-ticular applications. To facilitate further editing and customization,we also propose an interactive autocomplete authoring interface[Hsu et al. 2020; Xing et al. 2014] built upon our synthesis algorithmcomponents. Similar to existing design tools, users can create vari-ous free-style patterns. When they have sufficient exemplars andwould like to reduce further manual repetitions, they can specifyan output domain to be automatically filled [Kazi et al. 2012; Xinget al. 2014]. The synthesized patterns resemble and seamlessly con-nect with what has already been drawn. If not satisfied, users canaccept or modify the predictions, or ask for re-synthesis to maintainfull control. They can further designate specific source regions forcloning to target regions.We analyze our algorithm via ablation studies, compare it withalternative methods, and demonstrate the quality and accessibilityof our system via pattern design results. We plan to share our codealong with the publication of this paper to facilitate reproduction.In sum, the contributions of this work are: • A hierarchical representation and a synthesis method for bothgeometry and topology of continuous and discrete patterns. • An interactive authoring system with autocomplete functionsto reduce manual workloads and facilitate user control.
Our work is inspired by prior art in vector patterns, image textures,and interactive workflows. Procedural methods [Loi et al. 2017;Pedersen and Singh 2006; Santoni and Pellacini 2016] can produceintricate structures, but are limited in scope and difficult to gen-eralize for different types of patterns. Example-based methods aregeneral, but existing work predominantly focuses on image textures[Gatys et al. 2016; Lu et al. 2014; Wei et al. 2009] rather than vectorpatterns. Below, we survey methods most related to our work.
Example-based methods are designed to generate large patternsfrom small exemplars with an optional control provided by theusers [Barla et al. 2006; Bhat et al. 2004; Hsu et al. 2018; Hurtut et al.2009; Ijiri et al. 2008; Landes et al. 2013; Ma et al. 2013, 2011; Roveriet al. 2015; Tu et al. 2019; Zhou et al. 2007, 2006]. However, thesemethods target discrete elements or samples [Hsu et al. 2018, 2020;Hurtut et al. 2009; Ijiri et al. 2008; Landes et al. 2013; Ma et al. 2011;Tu et al. 2019] and treat continuous structures as special cases viacurve/surface reconstruction from point samples [Ma et al. 2011;Roveri et al. 2015]. Roveri et al. [2015] reconstructs the outputsurface from synthesized point samples via their associated surfacenormals without considering sample connections, and thus canonly be applied to surfaces relatively smooth to the underlyingsampling density. Tu et al. [2019] extends neural point synthesis bytreating a graph edge as a line of points, which is essentially pointsynthesis. The neural optimization method does not provide thesame flexibility and efficiency as in our method and it is unstablewhen the points contain attributes beyond positions. Relatively fewworks focus on curves, such as enriching details of given coarsecurves [Hertzmann et al. 2002] or growing L-system-like curves[Merrell and Manocha 2010]. Our system is inspired by these priorsample-based algorithms [Hsu et al. 2020; Ma et al. 2011; Roveri
ACM Trans. Graph., Vol. 39, No. 6, Article 168. Publication date: December 2020. ontinuous Curve Textures • 168:3
Fig. 3.
User Interface.
Our interface has a widget panel and a canvas. Thewidget panel provides basic tools, such as selection in a vector editor, aswell as controls to the graphics parameters, including color and pen width,and modes unique to our autocomplete system. et al. 2015], but we explicitly incorporate both samples and theirconnectivity into our representation and optimization to synthesizemore general continuous structures, as shown in Figure 2.
Workflow analysis has been investigated to assist various contentcreation task [Nancel and Cockburn 2014]. Examples includes staticand animated sketches [Xing et al. 2014, 2015], 3D sculpting [Penget al. 2020, 2018], texture design [Suzuki et al. 2017], hand-writingbeautification [Zitnick 2013], and image editing [Chen et al. 2011;Koyama et al. 2016]. Our work is inspired by prior autocomplete[Peng et al. 2018; Suzuki et al. 2017; Xing et al. 2014, 2015] andinteractive systems [Bian et al. 2018; Hsu et al. 2020; Kazi et al. 2012].However, these systems can automate only relatively simple patterns(e.g., repetitive hatches or strokes). Our method can automaticallygenerate diverse and complex continuous curve patterns to facilitateiterative design with reduced input workload.
Our system can be used for both automatic synthesis and interactiveediting. Similar to prior work like traditional texture synthesis [Weiet al. 2009], the user provides an exemplar pattern and lets oursystem automatically produce the output with desired size andshape. The exemplar is represented via Bézier curves. Inputs inother formats can be converted to Bézier curves (for example, byvectorizing a raster image).Since the automatic synthesis results might not be what the userswant and they might need to create new patterns manually, wealso provide an interface built upon our automatic synthesis al-gorithms for users to author patterns interactively. Through theinterface, users can specify an output domain in desired size andshape (Figure 4a) and let our system predict patterns that resemblewhat the users have already drawn ( autocomplete mode, Figures 4aand 4b). The users can also explicitly control the prediction bycopying-pasting from an input region to an output region ( clone mode, Figures 4c and 4d). They can accept, partially accept, or re-ject the predictions via keyboard shortcuts and mouse selections. (a) Autocomplete: before (b) Autocomplete: after
Source Target (c) Clone: before (d) Clone: after (e) User editFig. 4.
Autocomplete and clone.
In the autocomplete mode, the user canspecify an output region (shown in yellow) (a) and let our system generatepredicted patterns (b). In the clone mode, the user can specify a sourceregion (in red) and clone it to a target region (c). Our system can generatepredictions adaptive to the existing patterns (d), upon which users canperform further refinements (e). (e) is generated by editing the top leftcorner (in blue) of the predictions, including 1) partially rejecting severalpaths, 2) copy-pasting two elements, and 3) adding a path.
They can also perform further edits, such as selecting regions forre-synthesis or adding paths in the predictions. Please refer to thesupplementary video for live actions. Since continuous patterns of-ten contain complex structures beyond fine-grained autocomplete ofindividual strokes [Xing et al. 2014], we design our current interfaceto focus on autocomplete pattern regions instead of strokes.
Our method extends the sample-based element texture synthesismethod in [Ma et al. 2011] to consider not only individual pointsamples but also their curved connections via graphs [Hsu et al.2018, 2020]. We describe our pattern representation in Section 4.1,similarity measures in Section 4.2, and the corresponding synthesisand reconstruction algorithms in Sections 4.3 and 4.4. Our methodcan handle discrete elements, continuous structures, and their com-binations. We will describe when and how our algorithms treat themsimilarly or differently.
We represent patterns and elements via point samples and graphs(Figure 5) [Hsu et al. 2018; Ma et al. 2011; Roveri et al. 2015]. Eachsample 𝑠 records its position p ( 𝑠 ) , attributes a ( 𝑠 ) (Table 1), and 𝑖 ( 𝑠 ) ∈ [ , ] to indicate the confidence of its existence to optimizethe number of samples: u ( 𝑠 ) = ( p ( 𝑠 ) , a ( 𝑠 ) , 𝑖 ( 𝑠 )) . (1) Discrete elements.
Following [Ma et al. 2013, 2011], for discreteelements, sample attributes include a sample id 𝑞 ( 𝑠 ) that indicatesthe uniqueness of 𝑠 to other samples within its containing element. ACM Trans. Graph., Vol. 39, No. 6, Article 168. Publication date: December 2020. (a) Single-res (b) Multi-res (c) Pattern (d) GraphFig. 5.
Pattern representation.
Existing algorithms [Ma et al. 2011] only usea single-resolution element representation (a). We propose a hierarchicalelement representation to improve the synthesis quality (b). Our synthesisproceeds from red, blue, to yellow samples in coarse to fine levels. A pattern(c) is represented by a graph (d), where we record connections (yellow edges)and local path orientations (arrows) in addition to point samples. The colorof a sample indicates the number of connections | E ( 𝑠 ) | associated with it;orange, green, and blue indicate 1, 2, and 3.Table 1. Sample attributes.
Each sample 𝑠 records its attributes a ( 𝑠 ) that mayvary in terms of types of patterns (discrete element or continuous structure). 𝑞 ( 𝑠 ) ≥ indicates uniqueness of a sample relative to other samples withina discrete element, and 𝑞 ( 𝑠 ) = − for continuous structures. E ( 𝑠 ) = { 𝑒 𝑠𝑠 ′ } records all edges associated with 𝑠 , where 𝑒 𝑠𝑠 ′ is an edge between 𝑠 and 𝑠 ′ . o ( 𝑠 ) records the local orientations of the paths intersecting at sample 𝑠 . Attributes
Sample id Connectivity Orientation a ( 𝑠 ) 𝑞 ( 𝑠 ) E ( 𝑠 ) = { 𝑒 𝑠𝑠 ′ } o ( 𝑠 ) Continuous structures.
In discrete element synthesis, it is sufficientto use only samples with id 𝑞 ( 𝑠 ) (Figures 5a and 5b) to encodeelement shape because every element has the same topology [Maet al. 2011]. On the other hand, continuous structures are composedof paths and have more flexibilities. In our paper, we represent pathsby linear and quadratic Bézier curves, even though other vectorcurve formats can be easily added. The paths may be connected toeach other with complex topologies (Figure 1). Thus, samples aloneare not sufficient to disambiguate matching and reconstruction ofcontinuous structures. Therefore, we also consider connectivityamong samples, leading to a graph-based representation (Figure 5d).Note the method in [Hsu et al. 2018, 2020] also adopts a graph-basedrepresentation, but it is for discrete elements only without explicitlymodeling paths in a continuous pattern.Specifically, we record the connectivity E ( 𝑠 ) for 𝑠 within a ( 𝑠 ) . E ( 𝑠 ) = { 𝑒 𝑠𝑠 ′ } is the set of edges associated with 𝑠 , where 𝑒 𝑠𝑠 ′ repre-sents the edge between the two samples 𝑠 and 𝑠 ′ . We use 𝑖 ( 𝑒 ) ∈ [ ] to indicate the confidence of an edge existence; 𝑖 ( 𝑒 𝑠𝑠 ′ ) = / 𝑠 and 𝑠 ′ . We will relaxthe binary 𝑖 ( 𝑒 𝑠𝑠 ′ ) to be within the range [ ] during optimization-based synthesis. While E ( 𝑠 ) records pattern topology, we also recordthe tangent angles at 𝑠 on a path via an orientation attribute o ∈ R 𝑁 o as part of a ( 𝑠 ) , where 𝑁 o is the number of entries in o ( 𝑠 ) . Each entry 𝑜 of o is within [ , 𝜋 ) . We record o to facilitate pattern reconstruc-tion from graphs (Section 4.4.2 and fig. 15). Note that, for any inputsamples 𝑠 𝑖 , we always have | E ( 𝑠 𝑖 )| = 𝑁 o ( 𝑠 𝑖 ) (Figure 5d), where | E | is the size of the edge set E . However, this strict constraint is relaxedfor the output during the synthesis to facilitate faster convergence. We adopt a multi-resolutionrepresentation of sample graphs to handle patterns with complexstructures, analogous to prior multi-resolution algorithms for colortexture synthesis [Wei and Levoy 2000, 2001]. The representation issparser with less samples at coarser resolutions and becomes denserwith more samples at finer resolutions. By default, we use threelevel of hierarchies, which suffice in our experiments. Users candecide to use fewer levels if needed.
Discrete elements.
We generate element samples using a simpleapproach (Figures 5a and 5b). The finest level of samples are gener-ated by sampling the element polygon. The coarest level containsonly one sample centered at each element. The middle level of sam-ples are located at the midpoints of each downsampled finest-levelsamples and the coarest-level element centers.
Continuous structures.
For continuous patterns (Figure 5c), wesample the intersections (blue samples in Figure 5d) and ends ofpaths (orange samples) and uniformly place samples (green samples)along paths with spacing 𝛿 . We discuss the parameter 𝛿 values inSection 5.1. A core part of pattern synthesis is a measure of similarity betweenlocal regions [Ma et al. 2011; Roveri et al. 2015; Wei et al. 2009].Here, we describe our similarity measure for continuous patternsvia their sample-graph representation (Section 4.1), which, in turn,will form the basis for our synthesis optimization (Section 4.3).
The difference between two samples 𝑠 and 𝑠 ′ , which includes the differences in the global position p andattributes a , is defined as follows:ˆ p ( 𝑠, 𝑠 ′ ) = p ( 𝑠 ) − p ( 𝑠 ′ ) , (2)ˆ a ( 𝑠, 𝑠 ′ ) = (cid:16) ˆ 𝑞 ( 𝑠, 𝑠 ′ ) , ˆ E ( 𝑠, 𝑠 ′ ) (cid:17) (3)The differences in sample id attribute 𝑞 isˆ 𝑞 ( 𝑠, 𝑠 ′ ) = (cid:8) 𝑞 ( 𝑠 ) ≠ 𝑞 ( 𝑠 ′ ) (cid:9) , (4)where ( · ) is an indicator function that equals to one if its condition · holds, and zero otherwise. The edge set difference isˆ E ( 𝑠, 𝑠 ′ ) = (cid:169)(cid:173)(cid:171) ∑︁ 𝑒 𝑠 ˆ 𝑠 ∈ E ( 𝑠 ) 𝑑𝑖𝑠𝑡 ( 𝑒 𝑠 ˆ 𝑠 , 𝑒 𝑠 ′ ˆ 𝑠 ′ ) (cid:170)(cid:174)(cid:172) + 𝛽 (cid:12)(cid:12) | E ( 𝑠 )| − | E ( 𝑠 ′ )| (cid:12)(cid:12) , (5)where 𝑑𝑖𝑠𝑡 ( 𝑒 𝑠 ˆ 𝑠 , 𝑒 𝑠 ′ ˆ 𝑠 ′ ) = ∥ ˆ p ( 𝑠, ˆ 𝑠 ) − ˆ p ( 𝑠 ′ , ˆ 𝑠 ′ )∥ is the difference between 𝑒 𝑠 ˆ 𝑠 and 𝑒 𝑠 ′ ˆ 𝑠 ′ , and 𝑒 𝑠 ′ ˆ 𝑠 ′ = 𝑚 𝑒 ( 𝑒 𝑠 ˆ 𝑠 ) ∈ E ( 𝑠 ′ ) is the matching edgefor 𝑒 𝑠 ˆ 𝑠 via the Hungarian algorithm (which solves the one-to-onematching relationship between edges) [Kuhn 1955] to minimize thefirst term in Equation (5) ( 𝑚 indicates matching relationship). 𝛽 isa weighting parameter set to sampling distance 𝛿 (Section 4.1.1) inour experiments.For newly added output samples that do not have any edges,either by initialization or existence assignment (Section 4.3), wewant them to be useful and connected to existing output samples.To this end, they should be encouraged (via lower cost) to matchwith input samples during the search step (Section 4.3.3). We set ACM Trans. Graph., Vol. 39, No. 6, Article 168. Publication date: December 2020. ontinuous Curve Textures • 168:5 𝛽 = o within the attribute similarity term (Equa-tion (3)) since E already contain similar information in o . How-ever, we still need to update o during synthesis (assignment step,Section 4.3.4) and reconstruction (Section 4.4). This requires us tomatch orientation entries 𝑜 within o from 𝑠 and 𝑠 ′ respectively. Thematching 𝑜 ( 𝑠 ′ ) = 𝑚 𝑜 ( 𝑜 ( 𝑠 )) ∈ o ( 𝑠 ′ ) is computed via the Hungarianalgorithm [Kuhn 1955] by minimizing the sum of smallest absolutedifferences between matched 𝑜 ( 𝑠 ) and 𝑜 ( 𝑠 ′ ) ˆ o ( 𝑠, 𝑠 ′ ) = ∑︁ 𝑜 ( 𝑠 ) ∈ o ( 𝑠 ) ˆ 𝑜 ( 𝑠, 𝑠 ′ ) , (6)where ˆ 𝑜 ( 𝑠, 𝑠 ′ ) = min (| 𝑜 ( 𝑠 ) − 𝑚 𝑜 ( 𝑜 ( 𝑠 )) | , 𝜋 − | 𝑜 ( 𝑠 ) − 𝑚 𝑜 ( 𝑜 ( 𝑠 )) |) .We also have not found it necessary to include 𝑖 in the samplesimilarity measure (Equation (3)). Instead, 𝑖 will be used to optimizethe number of samples. We define n ( 𝑠 ) , the neighborhoodof 𝑠 , as a set of samples around 𝑠 ’s spatial vicinity within a certainradius 𝑟 . The neighborhood similarity is defined as ∥ n ( 𝑠 𝑜 ) − n ( 𝑠 𝑖 )∥ = ∑︁ 𝑠 ′ 𝑜 ∈ 𝑚 𝑛 ( n ( 𝑠 𝑖 )) ˆ u 𝑠 𝑜 𝑠 𝑖 ( 𝑠 ′ 𝑜 , 𝑠 ′ 𝑖 ) + ∑︁ 𝑠 ′ 𝑜 ∈ n ( 𝑠 𝑜 ) ⊖ 𝑚 𝑛 𝑐 ( 𝑠 ′ 𝑜 ) , (7)where ˆ u 𝑠 𝑜 𝑠 𝑖 ( 𝑠 ′ 𝑜 , 𝑠 ′ 𝑖 ) = ∥ ˆ p ( 𝑠 𝑜 , 𝑠 ′ 𝑜 ) − ˆ p ( 𝑠 𝑖 , 𝑠 ′ 𝑖 )∥ + 𝛾 ∥ ˆ a ( 𝑠 ′ 𝑜 , 𝑠 ′ 𝑖 )∥ (8)is the sample similarity between 𝑠 ′ 𝑜 and 𝑠 ′ 𝑖 within neighborhoodscentered at 𝑠 𝑜 and 𝑠 𝑖 respectively. 𝑠 ′ 𝑖 = 𝑚 𝑠 ( 𝑠 ′ 𝑜 ) is the matching inputsample for 𝑠 ′ 𝑜 . We discuss how to match samples within n ( 𝑠 𝑜 ) and n ( 𝑠 𝑖 ) in Section 4.2.3. The positional differences ∥ ˆ p ( 𝑠 𝑜 , 𝑠 ′ 𝑜 )− ˆ p ( 𝑠 𝑖 , 𝑠 ′ 𝑖 )∥ are computed in local neighborhood coordinate systems centeredat 𝑠 𝑜 or 𝑠 𝑖 . The two terms in Equation (7) partition n ( 𝑠 𝑜 ) into twosets. In the first term of Equation (7), 𝑚 𝑛 ( n ( 𝑠 𝑖 )) is the subset of n ( 𝑠 𝑜 ) matched with samples within n ( 𝑠 𝑖 ) . In the second term ofEquation (7), 𝑐 ( 𝑠 ′ 𝑜 ) is the cost resulting from unmatched outputsamples 𝑠 ′ 𝑜 . In our implementation, 𝛾 = .
5. Equation (7) is designedfor our robust neighborhood matching, described next.
In [Ma et al. 2011], eachoutput sample is forced to match with another sample in the in-put, which could be problematic since some output samples areoutliers and should not be matched to any input samples. Someoutput samples might be missing in the current iteration of opti-mization. But this forced matching allows to easily define samplesimilarity for various sample attributes, as shown in Section 4.2.1,since we have one-to-one sample correspondence. We call this hardneighborhood matching (Figure 6a). In [Roveri et al. 2015], the neigh-borhoods are matched via comparing their density fields estimatedwith Gaussian kernels. This similarity criterion is computed withthe neighborhoods as a whole.
There is no one-to-one correspondencebetween samples . We call it soft neighborhood matching (Figure 6b).The method [Roveri et al. 2015] "smears the sample attributes intotheir neighborhood" by encoding them as the height of the densitykernel, which could unnecessarily couple the position and attribute information. It is not easy to integrate soft matching with varioussample attributes, which can include edges.Instead, we propose to use a robust neighborhood matching thatexplicitly considers outliers in the output to address these issues(Figure 6c). An output sample is either matched with an input sampleor unmatched as an outlier with additional cost 𝑐 . We apply theHungarian algorithm to compute the matchings between input n ( 𝑠 𝑖 ) and output neighborhoods n ( 𝑠 𝑜 ) . The input of the Hungarianalgorithm is a cost matrix where each entry indicates the matchingcost between an output and an input sample. Inspired by [Riesenand Bunke 2009], we define our cost matrix C ∈ R 𝑁 n 𝑜 ×( 𝑁 n 𝑖 + 𝑁 n 𝑜 ) as: C = (cid:2) C 𝑚 C 𝑢 (cid:3) (9) C 𝑚 = ˆ u 𝑠 𝑜 𝑠 𝑖 ( 𝑠 ′ 𝑜 , 𝑠 ′ 𝑖 ) ˆ u 𝑠 𝑜 𝑠 𝑖 ( 𝑠 ′ 𝑜 , 𝑠 ′ 𝑖 ) · · · ˆ u 𝑠 𝑜 𝑠 𝑖 ( 𝑠 ′ 𝑜 , 𝑠 ′ 𝑁 n 𝑖 𝑖 ) ˆ u 𝑠 𝑜 𝑠 𝑖 ( 𝑠 ′ 𝑜 , 𝑠 ′ 𝑖 ) ˆ u 𝑠 𝑜 𝑠 𝑖 ( 𝑠 ′ 𝑜 , 𝑠 ′ 𝑖 ) · · · ˆ u 𝑠 𝑜 𝑠 𝑖 ( 𝑠 ′ 𝑜 , 𝑠 ′ 𝑁 n 𝑖 𝑖 ) ... ... . . . ... ˆ u 𝑠 𝑜 𝑠 𝑖 ( 𝑠 ′ 𝑁 n 𝑜 𝑜 , 𝑠 ′ 𝑖 ) ˆ u 𝑠 𝑜 𝑠 𝑖 ( 𝑠 ′ 𝑁 n 𝑜 𝑜 , 𝑠 ′ 𝑖 ) · · · ˆ u 𝑠 𝑜 𝑠 𝑖 ( 𝑠 ′ 𝑁 n 𝑜 𝑜 , 𝑠 ′ 𝑁 n 𝑖 𝑖 ) (10) C 𝑢 = 𝑐 𝑐 · · · 𝑐 𝑐 𝑐 · · · 𝑐 ... ... ... ...𝑐 𝑁 n 𝑜 𝑐 𝑁 n 𝑜 · · · 𝑐 𝑁 n 𝑜 , (11)where the superscripts of 𝑠 ′ 𝑜 or 𝑠 ′ 𝑖 represent the index of a samplewithin n ( 𝑠 𝑜 ) or n ( 𝑠 𝑖 ) . There are 𝑁 n 𝑖 and 𝑁 n 𝑜 samples within n ( 𝑠 𝑖 ) and n ( 𝑠 𝑜 ) , respectively. In [Ma et al. 2013], the sample matching iscomputed via only the C 𝑚 part of C , in which case every outputsample should be matched. Our cost matrix is augmented withthe C 𝑢 side, where each entry represents the cost of unmatchedoutliers in n ( 𝑠 𝑜 ) . We make sure there are enough samples in theinput neighborhood so that an output sample would not be matchedonly when it is an outlier that would result in a high cost increasein matching. For the same reason, we do not take missing outputsamples into account in the cost matrix formulation.In our implementation, 𝑐 𝑘 is set as min ( , . + . | E ( 𝑠 𝑘𝑜 )|) 𝛿 if theoutput sample 𝑠 𝑘𝑜 is from a continuous pattern, and 1 . × averagenearest neighbor distance if 𝑠 𝑘𝑜 is from a discrete element. In aninteractive system, we may synthesize predictions near the providedexemplars. If 𝑠 𝑘𝑜 is from the exemplars, 𝑐 𝑘 = ∞ because none of thesamples from the exemplars are outliers and all of them should bematched.In a neighborhood, there might be samples from both discreteelements and continuous structures. We only match samples fromthe same type of patterns and with the same id, i.e. continuousstructures only match with continuous structures (negative id), anddiscrete elements only match the same discrete elements and theirsamples with the same (non-negative) ids. Based on our pattern representation (Section 4.1) and similaritymeasures (Section 4.2), we now describe how to synthesize an outputsimilar to a given input.
ACM Trans. Graph., Vol. 39, No. 6, Article 168. Publication date: December 2020.
Output Input (a) Hard neighborhood matching (Density) (Density)(Kernel)
Output Input (b) Soft neighborhood matching
Output Input (c) Robust neighborhood matchingFig. 6.
Different neighborhood matching methods.
In (a) each sample in the output (left) is forced to match with another one in the input (right), which could beproblematic since some output samples are outliers and should not be matched to any input samples, and some output samples might be missing in thecurrent iteration of optimization. Matched samples are with the same color. Black indicates unmatched. In (b) samples are not explicitly matched but theneighborhood is matched as a whole, by transforming the samples with density kernels [Roveri et al. 2015]. There is no one-to-one correspondence betweensamples, and thus it is not easy to integrate with various sample attributes. Instead, we propose robust neighborhood matching (c) to generate high-qualitysample distributions to accommodate various types of patterns. We allow output samples to be unmatched if it can result in high matching cost increase. Inthe above example, the yellow output sample in (a) destroys the hard neighborhood matching and forces other (orange, and gray) samples to be less matched,while our robust matching leaves the bad sample unmatched.
We synthesize output predictions O via optimizing the following objective: 𝐸 (O) = ∑︁ 𝑠 𝑖 = 𝑚 ( 𝑠 𝑜 ) ,𝑠 𝑜 ∈O ∥ n ( 𝑠 𝑜 ) − n ( 𝑠 𝑖 )∥ + Θ (O , D) . (12)where 𝑠 𝑜 is matched with 𝑠 𝑖 . This energy sums up the similaritybetween every n ( 𝑠 𝑜 ) in O and its most similar n ( 𝑠 𝑖 ) via Equation (7). Θ (O , D) is the domain constraint term [Dumas et al. 2018] to en-courage the synthesized samples to stay within the user-specifieddomain D .The pattern optimization framework adopts an EM-like strategyto minimize Equation (12), by iterating the search and assignmentsteps as detailed below. Similar to prior patch-based texture synthe-sis methods [Efros and Freeman 2001; Liang et al. 2001], we copynew patches one-by-one with similar boundary patterns to existingpatches for initialization. Each next patch is selected to ensure highsimilarity (as evaluated by Equation (7)) in the overlapped boundaryregions with existing patches. In the overlapping regions, we onlycopy samples unmatched with any sample in the existing patches.We make sure that the initialized samples are within the output do-main D by removing samples outside it. We copy discrete elementsin wholes like [Ma et al. 2011]. In Section 5, we will show the ro-bustness of our method to random sample initializations (Figure 14).But patch-based initialization makes the algorithm converge faster,contributing to the responsiveness of the interface. For simplicity,we do not copy edges in the initialization step. We adopt PatchMatch [Barnes et al. 2009; Chenet al. 2012] to compute approximate nearest neighbors (ANN) foreach output sample. The standard PatchMatch algorithm 1) ran-domly generates the initial nearest neighbor field, and 2) alternatesbetween propagation and search steps by traversing the regularimage grid in a scanline order. Initially, we generate the ANN byrandomly assigning an output sample to an input sample (with iden-tical sample id for discrete elements). One issue is how to choose a sample traversal order. We follow the steps from [Chen et al. 2012]which works on meshes. We build a simple graph by connectingeach sample with its 𝑘 -nearest neighbors ( 𝑘 = 𝑂 ( 𝑛 O 𝑁 n ) ,where 𝑛 O is the total number of output samples and 𝑁 n is the av-erage number of samples within neighborhoods. Please refer toAppendix C for more details. Here, we describe how to determine thevalues of sample positions p , attributes including edge E and ori-entation o , as well as sample existence 𝑖 . The assignments of thesedifferent quantities are extended from the assignment step of pixelcolors [Kwatra et al. 2005] and sample positions [Ma et al. 2011] bytaking votes from overlapping output neighborhoods at the sameentity (such as sample or edge). In particular, discrete samples onlyhave sample id attributes 𝑞 , which is used in the search step to makesure only samples with the same 𝑞 are matched. Thus, only positionassignment is deployed for discrete samples. Position assignment.
For each output sample 𝑠 𝑜 and its neighbor 𝑠 ′ 𝑜 , there is a set of matched input sample pairs (cid:0) 𝑠 𝑖 , 𝑠 ′ 𝑖 (cid:1) provided bythe previous search step. The estimated distance ˆ p ( 𝑠 𝑜 , 𝑠 ′ 𝑜 ) between ACM Trans. Graph., Vol. 39, No. 6, Article 168. Publication date: December 2020. ontinuous Curve Textures • 168:7 output input (a) Matched neighborhood pair 1 output input (b) Matched neighborhood pair 2 output input (c) Matched neighborhood pair 3 output i ( ) = !" i ( ) = (d) Assign existenceFig. 7. Existence assignment example.
We visualize how to compute confidences of existence of output samples 𝑖 ( 𝑠 𝑜 ) (Equation (15) and algorithm 1) from a setof matched input and output neighborhoods. (a) (b) (c) show three pairs of matched input and output neighborhoods centered at different samples (shown inred) over the same set of samples. Matched samples are in the same color. Empty black circles indicate samples outside a neighborhood. Solid black circlesindicate samples within a neighborhood but unmatched. The black cross sample in (d) has 𝑖 ( 𝑠 𝑜 ) = , since it is unmatched with any 𝑠 𝑖 ( 𝑖 ( 𝑠 𝑖 ) = ) in (a) and(b). The green sample in (d) has confidence 𝑖 ( 𝑠 𝑜 ) = : in the three pairs of neighborhoods, there are two pairs (a) and (c) where each has an unmatched inputsample, which indicates there could be a missing sample in the output located at approximately the same location relative to its neighborhood center; the twounmatched input samples are merged to generate the green output sample in (d). 𝑒 ! 𝑒 " 𝑒 𝑒 $ s s s s (a) Initialization 𝑒 ! 𝑒 " 𝑒 𝑒 $ 𝐸 %& (𝐢(𝑒 ! ) = 1) < 𝐸 %& (𝐢(𝑒 ! ) = 0)(0.9 < 3.9) s s s s (b) Step 1 𝑒 ! 𝑒 " 𝑒 𝑒 $ 𝐸 %& (𝐢(𝑒 " ) = 1) < 𝐸 %& (𝐢(𝑒 " ) = 0)(1 < 3.4) s s s s (c) Step 2 𝑒 ! 𝑒 " 𝑒 𝑒 $ 𝐸 %& (𝐢(𝑒 ) = 1) > 𝐸 %& (𝐢(𝑒 ) = 0)(1.75 > 1.65) s s s s (d) Step 3 𝑒 ! 𝑒 " 𝑒 𝑒 $ 𝐸 %& (𝐢(𝑒 $ ) = 1) < 𝐸 %& (𝐢(𝑒 $ ) = 0)(1 < 2.2) s s s s (e) Step 4Fig. 8. Edge assignment example.
This example illustrates how to solve Equation (16) for edge assignment by solving Equation (21) in a loop. Each step(b)-(e) solves Equation (21) once. There are four samples and four potential edges (with 𝑖 ( 𝑒 ) > ). The numbers near edges or samples indicate theexpected confidence of existence 𝑖 ( 𝑒 ) of the edges or the expected number of edges | E ( 𝑠 ) | associated with samples (e.g. 𝑖 ( 𝑒 ) = . , | E ( 𝑠 ) | = . ).In (a), we initialize all 𝑖 ( 𝑒 ) = and sort all potential edges by its 𝑖 from the largest to the smallest: 𝑒 is the one with the largest 𝑖 and 𝑒 is the onewith the smallest. The optimization loops from 𝑒 (b) to 𝑒 (e) in decreasing 𝑖 ( 𝑒 ) values. The number below each 𝐸 ′ 𝑒 ( 𝑖 ( 𝑒 )) is its computed value (e.g. 𝐸 ′ 𝑒 ( 𝑖 ( 𝑒 ) = ) = | 𝑖 ( 𝑒 ) − 𝑖 ( 𝑒 ) | + | E ( 𝑠 ) − E ( 𝑠 ) | + | E ( 𝑠 ) − E ( 𝑠 ) | = | − | + | − | + | − . | = . ). The light green edges are not optimized with initialvalues 𝑖 ( 𝑒 ) = . The dark green edges are optimized with 𝑖 ( 𝑒 ) = . The red edge is optimized with 𝑖 ( 𝑒 ) = . The light red indicates there is no edge afteroptimization. 𝑠 𝑜 and 𝑠 ′ 𝑜 is ˆ p ( 𝑠 𝑜 , 𝑠 ′ 𝑜 ) ≈ p ( 𝑠 𝑖 ) − p ( 𝑠 ′ 𝑖 ) . (13)We use least squares [Ma et al. 2011] to estimate p ( 𝑠 𝑜 ) byarg min { p ( 𝑠 𝑜 ) } ∑︁ 𝑠 𝑜 ∈O ∑︁ 𝑠 ′ 𝑜 ∈ n ( 𝑠 𝑜 ) (cid:13)(cid:13) ˆ p ( 𝑠 𝑜 , 𝑠 ′ 𝑜 ) − (cid:0) p ( 𝑠 𝑖 ) − p ( 𝑠 ′ 𝑖 ) (cid:1)(cid:13)(cid:13) + ∑︁ 𝑠 𝑜 ∉ D ∥ ˆ p ( 𝑠 𝑜 , D)∥ . (14)The second term in Equation (14) is the domain constraint to en-courage output samples to stay within D , where ˆ p ( 𝑠 𝑜 , D) is theshortest vector from 𝑠 𝑜 to the boundary of D , and 𝑠 𝑜 ∉ D indicates 𝑠 𝑜 is outside D . Existence assignment.
Our method adjusts the number of sampleswithin local regions during the synthesis process for better quality.The number of samples is optimized via existence 𝑖 assignment,again via a voting scheme:arg min 𝑖 ( 𝑠 𝑜 ) ∈[ ] ∑︁ 𝑠 𝑖 ∈{ 𝑠 𝑖 } | 𝑖 ( 𝑠 𝑜 ) − 𝑖 ( 𝑠 𝑖 )| , (15)where 𝑠 𝑖 runs through the set { 𝑠 𝑖 } we collect during neighborhoodmatching, i.e. the corresponding input samples of an output sample from overlapping input neighborhoods. 𝑖 ( 𝑠 𝑖 ) = 𝑠 𝑜 is not matchedwith any 𝑠 𝑖 in a pair of matched input and output neighborhoods,and 𝑖 ( 𝑠 𝑖 ) = 𝑖 ( 𝑠 𝑜 ) ∈ [ ] of an output sample 𝑠 𝑜 . Every iteration, we remove output samples 𝑠 𝑜 whose 𝑖 ( 𝑠 𝑜 ) < .
5. The above as-signment step is applied to samples that are already in the outputsample distribution. To add back missing output samples, we firstgenerate candidate samples, merge them as output samples, andpick those with 𝑖 ( 𝑠 𝑜 ) > . Edge assignment.
We assign edges by optimizing the followingobjective:arg min { 𝑖 ( 𝑒 𝑠𝑜𝑠 ′ 𝑜 ) ∈{ , }} ∑︁ { 𝑒 𝑠𝑜𝑠 ′ 𝑜 } (cid:12)(cid:12) 𝑖 ( 𝑒 𝑠 𝑜 𝑠 ′ 𝑜 ) − 𝑖 ( 𝑒 𝑠 𝑜 𝑠 ′ 𝑜 ) (cid:12)(cid:12) + ∑︁ { 𝑠 𝑜 } (cid:12)(cid:12)(cid:12) | E ( 𝑠 𝑜 )| − | E ( 𝑠 𝑜 )| (cid:12)(cid:12)(cid:12) , (16) ACM Trans. Graph., Vol. 39, No. 6, Article 168. Publication date: December 2020. where the first term computes the difference between the actual andexpected edge confidences 𝑖 ( 𝑒 𝑠 𝑜 𝑠 ′ 𝑜 ) ∈ [ ] . 𝑖 is the vote by overlap-ping input neighborhoods on the same edge, computed using leastsquares by replacing samples in Equation (15) with edges. { 𝑒 𝑠 𝑜 𝑠 ′ 𝑜 } isthe set of edges that have 𝑖 ( 𝑒 𝑠 𝑜 𝑠 ′ 𝑜 ) >
0, and there is no edge between 𝑠 𝑜 and 𝑠 ′ 𝑜 if 𝑖 ( 𝑒 𝑠 𝑜 𝑠 ′ 𝑜 ) =
0. The second term computes the differencesbetween the optimized number of edges | E ( 𝑠 𝑜 )| and the expectednumber of edges | E ( 𝑠 𝑜 )| connected to 𝑠 𝑜 . | E ( 𝑠 𝑜 )| is similarly com-puted by voting from overlapping output neighborhoods on thesame sample 𝑠 𝑜 :arg min | E ( 𝑠 𝑜 ) | ∑︁ 𝑠 𝑖 ∈{ 𝑠 𝑖 } (cid:12)(cid:12)(cid:12) | E ( 𝑠 𝑜 )| − | E ( 𝑠 𝑖 )| (cid:12)(cid:12)(cid:12) . (17)Basically, we compute the average of {| E ( 𝑠 𝑖 )|} . In sum, the firstterm is edge-centric while the second is sample-centric.It is non-trivial to optimize Equation (16), where the optimizationvariables { 𝑖 ( 𝑒 𝑠 𝑜 𝑠 ′ 𝑜 )} are binary. Thus, we solve it in a greedy fash-ion. We initialize all 𝑖 ( 𝑒 𝑠 𝑜 𝑠 ′ 𝑜 ) =
0. We sort output edges { 𝑒 𝑠 𝑜 𝑠 ′ 𝑜 } byits expected confidence of existence 𝑖 ( 𝑒 𝑠 𝑜 𝑠 ′ 𝑜 ) , and optimize 𝑖 ( 𝑒 𝑠 𝑜 𝑠 ′ 𝑜 ) greedily by looping over the sorted { 𝑒 𝑠 𝑜 𝑠 ′ 𝑜 } in decreasing confidence.For each 𝑒 𝑠 𝑜 𝑠 ′ 𝑜 , we decide whether 𝑖 ( 𝑒 𝑠 𝑜 𝑠 ′ 𝑜 ) = { 𝑖 ( 𝑒 𝑠 𝑜 𝑠 ′ 𝑜 )} to a single edge 𝑖 ( 𝑒 𝑠 ∗ 𝑜 𝑠 ∗′ 𝑜 ) to be optimized and the rest, the univariateversion of Equation (16) can be written as:arg min 𝑖 ( 𝑒 𝑠 ∗ 𝑜𝑠 ∗′ 𝑜 ) ∈{ , } 𝐸 ′ 𝑒 + 𝐸 ′′ 𝑒 , (18)where 𝐸 ′ 𝑒 (cid:16) 𝑖 ( 𝑒 𝑠 ∗ 𝑜 𝑠 ∗′ 𝑜 ) (cid:17) = (cid:12)(cid:12) 𝑖 ( 𝑒 𝑠 ∗ 𝑜 𝑠 ∗′ 𝑜 ) − 𝑖 ( 𝑒 𝑠 ∗ 𝑜 𝑠 ∗′ 𝑜 ) (cid:12)(cid:12) + (cid:12)(cid:12)(cid:12) | E ( 𝑠 ∗ 𝑜 )| − | E ( 𝑠 ∗ 𝑜 )| (cid:12)(cid:12)(cid:12) + (cid:12)(cid:12)(cid:12) | E ( 𝑠 ∗′ 𝑜 )| − | E ( 𝑠 ∗′ 𝑜 )| (cid:12)(cid:12)(cid:12) , (19) 𝐸 ′′ 𝑒 (cid:16) 𝑖 ( 𝑒 𝑠 ∗ 𝑜 𝑠 ∗′ 𝑜 ) (cid:17) = ∑︁ { 𝑒 𝑠𝑜𝑠 ′ 𝑜 }⊖ 𝑒 𝑠 ∗ 𝑜𝑠 ∗′ 𝑜 (cid:12)(cid:12) 𝑖 ( 𝑒 𝑠 𝑜 𝑠 ′ 𝑜 ) − 𝑖 ( 𝑒 𝑠 𝑜 𝑠 ′ 𝑜 ) (cid:12)(cid:12) + ∑︁ { 𝑠 𝑜 }⊖{ 𝑠 ∗ 𝑜 ,𝑠 ∗′ 𝑜 } (cid:12)(cid:12)(cid:12) | E ( 𝑠 𝑜 )| − | E ( 𝑠 𝑜 )| (cid:12)(cid:12)(cid:12) . (20)Since 𝐸 ′′ 𝑒 is a constant in Equation (18), it is equivalent to:arg min 𝑖 ( 𝑒 𝑠 ∗ 𝑜𝑠 ∗′ 𝑜 ) ∈{ , } 𝐸 ′ 𝑒 . (21)Equation (21) can be solved with brute-force search. The searchspace is 2 ( { , } ). Figure 8 illustrates how to solve Equation (16) bysolving Equation (21) in a loop. Orientation assignment.
In the search step (Section 4.3.3), each o ( 𝑠 𝑜 ) is matched with a set of { o ( 𝑠 𝑖 )} associated with input samplescoming from different input neighborhoods, and each entry 𝑜 ( 𝑠 𝑜 ) ∈ o ( 𝑠 𝑜 ) has been matched with a 𝑜 ( 𝑠 𝑖 ) ∈ o ( 𝑠 𝑖 ) . The local orientationattribute o ( 𝑠 𝑜 ) is updated by a voting scheme among { o ( 𝑠 𝑖 )} , where { o ( 𝑠 𝑖 )} could have different lengths across different 𝑠 𝑖 .We optimize both dimension 𝑁 o and value of entries 𝑜 in order.In the input exemplar, the number of orientation entries 𝑁 o ( 𝑠 𝑖 ) o o o o o o o o o (a) { o ( 𝑠 𝑖 ) } o o (b) o ( 𝑠 𝑜 ) o = median( o , o , o ) o = ( o + o + o )/3 o = ( o + o + o )/3 (c) Updated (b)Fig. 9. Orientation assignment example.
This example illustrates the orien-tation assignment step when 𝑁 o ( 𝑠 𝑜 ) is increased from to . The outputsample (b) is matched with the three input samples (a). Matched orienta-tions 𝑜 (arrows) are visualized in the same colors. Unmatched inputs 𝑜 are inblack. (c) shows the updated orientations of the output sample. The orange 𝑜 new and green orientations 𝑜 new are updated by averaging matched inputorientations. The black output orientation 𝑜 new is newly added by choosingthe median from three unmatched orientations ( 𝑜 , 𝑜 , 𝑜 ) in (a). equals | E ( 𝑠 𝑖 )| . Thus 𝑁 o can be computed like in Equation (17) androunding the result as integers. Essentially, we are trying to findan integer 𝑁 o ( 𝑠 𝑜 ) that is the closest to the arithmetic average of { 𝑁 o ( 𝑠 𝑖 )} .Similarly, we can update the values 𝑜 ( 𝑠 𝑜 ) in o ( 𝑠 𝑜 ) using the samevoting scheme to Equations (15) and (17). A special case is when 𝑁 o ( 𝑠 𝑜 ) is updated to a new value (changing o ( 𝑠 𝑜 ) vector length). Inthis case, we will need to add or remove one or several entries to orfrom the original o ( 𝑠 𝑜 ) . To remove an entry from o ( 𝑠 𝑜 ) , we pick theone whose matched set of input votes { 𝑜 ( 𝑠 𝑖 )} has the largest vari-ance. (We have experimented with another strategy that removes 𝑜 ( 𝑠 𝑜 ) whose matched set of input votes { 𝑜 ( 𝑠 𝑖 )} has the least numberof entries, but have not found visible differences to the maximumvariance strategy above.) To add an entry to o ( 𝑠 𝑜 ) , we collect orienta-tion entries { 𝑜 ′ ( 𝑠 𝑖 )} from the input samples that remain unmatchedto any orientation entries o ( 𝑠 𝑜 ) of the matched output sample, andadd a new entry 𝑜 ( 𝑠 𝑜 ) into o ( 𝑠 𝑜 ) as the median from the unmatchedset { 𝑜 ′ ( 𝑠 𝑖 )} . An example is illustrated in Figure 9. In the rare casewhere we need to add more than one entry to o ( 𝑠 𝑜 ) , we randomlychoose from { 𝑜 ′ ( 𝑠 𝑖 )} after the median is used for the first add-on. Instead of using a single-resolutionrepresentation [Ma et al. 2011], we apply a hierarchical representa-tion (Section 4.1.1) for multi-resolution synthesis. We first synthesizethe predictions at a coarse level using sparse representation, andthen reconstruct the patterns based on sparse samples. We continuethis process with a denser and denser pattern representation. Fig-ure 5b shows an example of multi-resolution element representation.For continuous structures, the sampling distance 𝛿 of continuouspattern is gradually increasing with respect to the level of hierarchy.During synthesis, we use multi-scale neighborhood sizes to keepboth large and local structures. The neighborhood size is graduallyreduced at different hierarchies. In our implementation, at eachhierarchy, there are 7 search-assignment iterations. See Figure 10for an example. The reconstruction step takes a synthesized pattern representationas input to generate output patterns that may consist of discreteelements and continuous structures.
ACM Trans. Graph., Vol. 39, No. 6, Article 168. Publication date: December 2020. ontinuous Curve Textures • 168:9
Leaves level 1 level 2 level 3 e n e r gy
14 217 number of iterations
Exemplar Initial step First level Second level Third level level 1level 2 level 3number of iterations e n e r gy Energy plotFig. 10.
Hierarchical synthesis visualization.
The hierarchical synthesis proceeds from the coarsest (first level) to the finest levels (third level), with upsamplingsample representations (Figure 5b). The hierarchical synthesis gradually refines the pattern from large to small scale structures. Note that the initialization hasfewer elements than required (top row), hence our existence optimization adaptively controls the total number of elements.
For discrete elements, each sample is uniquelyassociated with an element. The reconstruction is to transform theelement shapes by treating samples as control points. Specifically,we assume similarity transform to reconstruct the elements.
We have synthesized a graph whosesample positions and edge connections represent the topology ofthe output (Section 4.3). However, these edges are piecewise lin-ear, and they thus capture only connectivity/topology, but notshape/geometry information. The original continuous patterns canbe composed of smooth paths (e.g. quadratic Bézier curves). There-fore, we need to reconstruct paths from the graph samples and edges.The samples are used as control points of Bézier curves.Next, we talk about how to identify which sample and whichedge are included within which path. This process relies on thesynthesized sample orientation attributes o .Samples with only one neighbor are unambiguous and thus onlyincluded within one path. Samples with only two neighbors couldbe included in one path (all blue samples with two neighbors inFigure 11) as path samples, or two paths as junction samples (e.g. theyellow sample in Figure 11a). Samples with more than two neighborsare junction samples (e.g. the red sample in Figure 11d) that areincluded in multiple potential paths.To disambiguate these cases, we examine a sample’s local orien-tations o ( 𝑠 ) . Since 𝑜 ∈ o ( 𝑠 ) should be tangent to the sample’s localpath, if a sample 𝑠 has a pair of orientations 𝑜 ( 𝑠 ) that are almostopposite (8 𝜋 / < absolute orientation difference < 𝜋 / 𝑜 ( 𝑠 ) (if any) that are opposite (e.g., a pair of arrowsassociated with 2-neighbor blue samples in Figures 11a and 11d, orthe orange and green ones associated with the red junction samplein Figure 11d). Second, we match local orientations 𝑜 ( 𝑠 ) with edges 𝑒 ∈ E ( 𝑠 ) connected to the sample using the Hungarian algorithmby minimizing the sum of absolute difference between local orienta-tion and edge angles. The arrows and graph edges in Figures 11aand 11d with the same colors are matched. Third, we generate a pathby including edges that are connected together and matched withopposite orientations. A Bézier curve is generated by interpolatingsamples along a path. This reconstruction strategy using o can helppreserving the original curve shapes, as demonstrated in Figure 15. We evaluate our method with sample results, ablation studies, andcomparisons with existing art. We will make our code repository[Tu 2020] public to facilitate future research.
Our method can automatically synthesize satisfactory results fora variety of patterns without user intervention, as exemplified inFigures 1, 12 and 13 and our (full) results in Figures 2, 10 and 14 to 16.However, like existing techniques, our method might not alwaysproduce what users would like to have, and some artifacts can bevisible in local regions (such as unfinished or dangling componentsin Figure 1a and Figure 2g or inconsistent curvatures in Figure 10kbottom) and global structures (such as the regular and warped gridsin Figures 12h and 12p, the rectangular blocks in Figure 12l, and thestraight lines in Figures 1e and 12t). For further quality improve-ment and customization, users can also interactively edit the systemsuggestions via our system interface, as demonstrated in Figure 13.Unless otherwise noted, all our results are produced with three hi-erarchies using neigborhood radii 𝑟 ∈ { , , } with samplingdistance 𝛿 ∈ { , , } , while the longer side of bounding box ofexemplars are varying between 250 and 500. Our method is robustto variations of neighborhood radii. See Appendix B for more detailsabout our parameter settings. ACM Trans. Graph., Vol. 39, No. 6, Article 168. Publication date: December 2020. (a) 2-neighbor 𝑠 (b) 2 paths from (a) (c) 1 path from (a) (d) > 𝑠 (e) Recon from (d) (f) Other possible reconstructions from (d)Fig. 11. Curve reconstruction from a graph using orientation attributes.
If we only consider the different statuses of the yellow sample in (a), there are twopossible reconstructions (b)(c), depending on whether it is a junction (b) or path (c) sample. If we only consider the different statuses of the red sample in (d),there are four possible reconstructions (e)(f), depending on the red sample is included within which two or three paths. For the yellow (in (a)) and red samples(in (d)), we decide the reconstruction by examining its associated local orientation attribute and the fact a pair of local orientations of a sample should beopposite if the sample is included within the path. Our algorithm will reconstruct (b) from (a) and (e) from (d).(a) Zentangle (b) (c) Diffusion (d) (e) Wet flow (f) (g) Brick wall (h)(i) Fence (j) (k) Blocks (l) (m) Heart (n) (o) Curve grid (p)(q) Isocontour (r) (s) Circuit (t) (u) Chain (v) (w) Fabric (x)Fig. 12.
Automatic synthesis results by our method.
Within each pair of images, the input exemplar is smaller and shown on the left, the automatic synthesisresult is bigger and shown on the right.
Although we use patch-based methods for initialization in our im-plementation, our algorithm is robust to different initial conditions(Figure 14), even if the initial sample distribution is randomly dis-tributed (white noise). Figure 15 is the ablation study for the ori-entation attribute o . Figure 16 shows other components of our al-gorithm. Without the edge term (Equation (5)) or robust matching (Section 4.2.3) in the search step, our algorithm produces lower qual-ity results with obvious artifacts. Without existence assignment (thethird paragraph in Section 4.3.4), the algorithm cannot automaticallyadjust the number of samples within local regions and can produceempty space or extra broken curves. To our knowledge, there is no previous example-based method thatcan generate the types of patterns we target. The sample-based
ACM Trans. Graph., Vol. 39, No. 6, Article 168. Publication date: December 2020. ontinuous Curve Textures • 168:11
Unevenbricks Auto User 16 + 11 Edited Zigzag Auto User 11 + 2 EditedPrisma Auto User 5 + 28 Edited Waves (small) Auto User 21 + 0 EditedFig. 13.
Automatic synthesis and user-assisted results.
We count the number of user operations needed for correcting artifacts. Red indicates the number ofrejection and blue the number of manual path drawing.VoronoiTree Patch-based initialization Patch-based final output Random sample initialization Random sample final outputFig. 14.
Algorithm robustness to different initialization.
Our algorithm can generate similar results with both patch-based or random initialization. methods in [Ma et al. 2011; Roveri et al. 2015; Tu et al. 2019] arethe most related. We compare against [Ma et al. 2011; Roveri et al.2015] and a state-of-art point distribution synthesis method in [Tuet al. 2019] which applies convolutional neural networks to preserveboth local and global structures. As shown in Figure 2, our methodcan produce better spatial sample distributions than [Ma et al. 2011;Roveri et al. 2015; Tu et al. 2019]. Note that we compare only sample distributions in Figure 2 since it is unclear how to reconstruct contin-uous curve patterns from synthesized samples without connectivity[Ma et al. 2011; Roveri et al. 2015; Tu et al. 2019].We also enhance [Ma et al. 2011] for comparisons, by incorpo-rating it with the sample connectivity (Figure 5d) and edge as-signment step (Equation (16)), but without the edge set difference(Equation (5)) and robust matching (Section 4.2.3) in the searchstep, as well as without the existence assignment step (the thirdparagraph in Section 4.3.4). As shown in Figure 17, our method
ACM Trans. Graph., Vol. 39, No. 6, Article 168. Publication date: December 2020. (a) Crocodile skin (b) With o (c) Zoom-in of (b)(d) Without o (e) Flame (f) With o (g) Zoom-in of (f)(h) Without o Fig. 15.
Ablation study for o . The orientation attribute o is useful to faithfully recover curve appearance in the exemplars. In the "crocodile skin" example, thecurves should be smooth at junction; in the "flame" example, the curves should be sharp at the flame tip.Fish scaleRoof tiles Without edge search Without robust matching Without existence assignment Full resultFig. 16. Ablation study.
Without using edges (Equation (5)) in the search step (the second column) or robust matching (Section 4.2.3) that considers outliers(the third column), the algorithm produces lower quality results. Without existence assignment (the third paragraph in Section 4.3.4), the algorithm producesbroken curves and empty space due to outliers and missing samples (the forth column). Our results are shown in the last column. can generate better results than the enhanced version of [Ma et al.2011]. Unlike for [Ma et al. 2011], we are unable to enhance [Roveriet al. 2015; Tu et al. 2019] due to the lack of one-to-one samplecorrespondences which are needed for the edge assignment step.
Repetitive patterns have many applications, whose creation hasbeen a main focus of research in computer graphics and interactivetechniques. This work focuses on methods and interfaces to helpusers author continuous curve patterns. Analysis and results ofdiverse patterns have demonstrated the promise of our approach.Like other neighborhood-based texture/pattern synthesis meth-ods, our algorithm also assumes local properties and thus cannotcapture global structures and may introduce stochastic variations, such as broken and distorted curves, as shown in Section 5.1. Theseartifacts can be reduced by other improvements, such as bidirec-tional similarity, additional feature masks, and smart initialization[Kaspar et al. 2015].Our current reconstruction algorithm is based on Bézier curveinterpolation, which might not preserve the exemplar curves. Onepossibility is to treat each curve segment like a discrete elementand reconstruct via sample-based warping [Hsu et al. 2020; Ma et al.2011], while ensuring that curve segments sharing common samplesare well connected.Our current algorithm treats discrete elements and continuousstructures separately and thus might not preserve identifiable el-ements within continuous structures, as exemplified in Figure 18.
ACM Trans. Graph., Vol. 39, No. 6, Article 168. Publication date: December 2020. ontinuous Curve Textures • 168:13
Waves (large) Roof tilesFish scale Enhanced [Ma et al. 2011] Ours Wood ring Enhanced [Ma et al. 2011] OursFig. 17.
Comparison of our algorithm with enhanced [Ma et al. 2011].
We compare our methods to an enhanced version of [Ma et al. 2011] that incorporates ourideas, including the sample connectivity and edge assignment.(a) Input (b) Output (c) Input (d) OuptutFig. 18.
Failure case.
Our algorithm fails to preserve the identifiable DNA-segment and tree-leaf elements within the continuous structures.
A potential future work is to find a unified representation and ap-proach for both discrete and continuous patterns.The pattern synthesis requires nearest neighborhood searchingfor output samples, which can become computationally expensivefor large outputs. This neighborhood searching process can be read-ily parallelized [Huang et al. 2007].We focus on curves as the first step to handle continuous vectorpatterns. A next step is to incorporate more vector graphics featuresas parts of the sample/edge attributes, such as color and thickness,as well as higher dimensional primitives including 2D regions and3D volumes [Takayama et al. 2010; Wang et al. 2011, 2010]. Morecontrols can also be added to facilitate more diverse authoring effectssuch as local variations in scales and orientations [Hsu et al. 2020].In addition to optimizing pattern appearance as in this work, addingmechanical structures constraints can facilitate the application ofcurve structures for rapid manufacturing [Bian et al. 2018; Chenet al. 2017, 2016; Li et al. 2019; Zehnder et al. 2016; Zhou et al. 2014].
ACKNOWLEDGMENTS
We would like to thank the anonymous reviewers for their valuablefeedback. Peihan Tu conducted parts of this research as a visitingstudent with the University of Tokyo and an intern with AdobeResearch. This work has been partially supported by an Adobe giftfunding and JSPS KAKENHI Grant Number 17H00752.
REFERENCES
Pascal Barla, Simon Breslav, Joëlle Thollot, François Sillion, and Lee Markosian. 2006.Stroke pattern analysis and synthesis. In
Computer Graphics Forum , Vol. 25. WileyOnline Library, 663–671.Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. 2009. Patch-Match: A Randomized Correspondence Algorithm for Structural Image Editing.
ACM Trans. Graph.
28, 3, Article 24 (July 2009), 11 pages. https://doi.org/10.1145/1531326.1531330Pravin Bhat, Stephen Ingram, and Greg Turk. 2004. Geometric texture synthesis byexample. In
SGP ’04 . 41–44.Xiaojun Bian, Li-Yi Wei, and Sylvain Lefebvre. 2018. Tile-based Pattern Design withTopology Control.
Proc. ACM Comput. Graph. Interact. Tech.
1, 1, Article 23 (July2018), 15 pages. https://doi.org/10.1145/3203204Hsiang-Ting Chen, Li-Yi Wei, and Chun-Fa Chang. 2011. Nonlinear Revision Controlfor Images.
ACM Trans. Graph.
30, 4, Article 105 (July 2011), 10 pages. https://doi.org/10.1145/2010324.1965000Weikai Chen, Yuexin Ma, Sylvain Lefebvre, Shiqing Xin, Jonàs Martínez, and wenpingwang. 2017. Fabricable Tile Decors.
ACM Trans. Graph.
36, 6, Article 175 (Nov. 2017),15 pages. https://doi.org/10.1145/3130800.3130817Weikai Chen, Xiaolong Zhang, Shiqing Xin, Yang Xia, Sylvain Lefebvre, and WenpingWang. 2016. Synthesis of Filigrees for Digital Fabrication.
ACM Trans. Graph.
35, 4,Article 98 (July 2016), 13 pages. https://doi.org/10.1145/2897824.2925911Xiaobai Chen, Tom Funkhouser, Dan B Goldman, and Eli Shechtman. 2012. Non-parametric texture transfer using meshmatch.
Adobe Technical Report arXiv preprint arXiv:1811.02626 (2018).Alexei A. Efros and William T. Freeman. 2001. Image Quilting for Texture Synthesisand Transfer. In
SIGGRAPH ’01 . 341–346. https://doi.org/10.1145/383259.383296Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2016. Image Style TransferUsing Convolutional Neural Networks. In
CVPR ’16 . 2414–2423.ACM Trans. Graph., Vol. 39, No. 6, Article 168. Publication date: December 2020.
Aaron Hertzmann, Nuria Oliver, Brian Curless, and Steven M. Seitz. 2002. CurveAnalogies. In
EGRW ’02 . 233–246.Chen-Yuan Hsu, Li-Yi Wei, Lihua You, and Jian Jun Zhang. 2018. Brushing ElementFields. In
SIGGRAPH Asia 2018 Technical Briefs (SA ’18) . Article 6, 4 pages. https://doi.org/10.1145/3283254.3283274Chen-Yuan Hsu, Li-Yi Wei, Lihua You, and Jian Jun Zhang. 2020. Autocomplete ElementFields. In
CHI ’20 . 1–13. https://doi.org/10.1145/3313831.3376248Hao-Da Huang, Xin Tong, and Wen-Cheng Wang. 2007. Accelerated parallel textureoptimization.
Journal of Computer Science and Technology
22, 5 (2007), 761–769.T. Hurtut, P.-E. Landes, J. Thollot, Y. Gousseau, R. Drouillhet, and J.-F. Coeurjolly. 2009.Appearance-guided Synthesis of Element Arrangements by Example. In
NPAR ’09 .51–60. https://doi.org/10.1145/1572614.1572623Takashi Ijiri, Radomír Mech, Takeo Igarashi, and Gavin Miller. 2008. An Example-basedProcedural System for Element Arrangement. In
Computer Graphics Forum , Vol. 27.Wiley Online Library, 429–436.Alexandre Kaspar, Boris Neubert, Dani Lischinski, Mark Pauly, and Johannes Kopf.2015. Self Tuning Texture Optimization.
Comput. Graph. Forum
34, 2 (May 2015),349–359. https://doi.org/10.1111/cgf.12565Rubaiat Habib Kazi, Takeo Igarashi, Shengdong Zhao, and Richard Davis. 2012. Vignette:Interactive Texture Design and Manipulation with Freeform Gestures for Pen-and-ink Illustration. In
CHI ’12 . 1727–1736. https://doi.org/10.1145/2207676.2208302Yuki Koyama, Daisuke Sakamoto, and Takeo Igarashi. 2016. SelPh: Progressive Learningand Support of Manual Photo Color Enhancement. In
CHI ’16 . 2520–2532. https://doi.org/10.1145/2858036.2858111Harold W Kuhn. 1955. The Hungarian method for the assignment problem.
Navalresearch logistics quarterly
2, 1-2 (1955), 83–97.Vivek Kwatra, Irfan Essa, Aaron Bobick, and Nipun Kwatra. 2005. Texture Optimizationfor Example-based Synthesis.
ACM Trans. Graph.
24, 3 (July 2005), 795–802. https://doi.org/10.1145/1073204.1073263Vivek Kwatra, Arno Schödl, Irfan Essa, Greg Turk, and Aaron Bobick. 2003. GraphcutTextures: Image and Video Synthesis Using Graph Cuts. In
SIGGRAPH ’03 . 277–286.https://doi.org/10.1145/1201775.882264Pierre-Edouard Landes, Bruno Galerne, and Thomas Hurtut. 2013. A Shape-AwareModel for Discrete Texture Synthesis.
Computer Graphics Forum
32, 4 (2013), 67–76.Yifei Li, David E. Breen, James McCann, and Jessica Hodgins. 2019. Algorithmic QuiltingPattern Generation for Pieced Quilts. In
Proceedings of the 45th Graphics InterfaceConference on Proceedings of Graphics Interface 2019 (GI& . Article 13, 9 pages.https://doi.org/10.20380/GI2019.13Lin Liang, Ce Liu, Ying-Qing Xu, Baining Guo, and Heung-Yeung Shum. 2001. Real-timeTexture Synthesis by Patch-based Sampling.
ACM Trans. Graph.
20, 3 (July 2001),127–150. https://doi.org/10.1145/501786.501787Hugo Loi, Thomas Hurtut, Romain Vergne, and Joelle Thollot. 2017. Programmable 2DArrangements for Element Texture Design.
ACM Trans. Graph.
36, 4, Article 105a(May 2017). https://doi.org/10.1145/3072959.2983617Jingwan Lu, Connelly Barnes, Connie Wan, Paul Asente, Radomir Mech, and AdamFinkelstein. 2014. DecoBrush: Drawing Structured Decorative Patterns by Example.
ACM Trans. Graph.
33, 4, Article 90 (July 2014), 9 pages. https://doi.org/10.1145/2601097.2601190Jingwan Lu, Fisher Yu, Adam Finkelstein, and Stephen DiVerdi. 2012. HelpingHand:Example-based Stroke Stylization.
ACM Trans. Graph.
31, 4, Article 46 (July 2012),10 pages. https://doi.org/10.1145/2185520.2185542Chongyang Ma, Li-Yi Wei, Sylvain Lefebvre, and Xin Tong. 2013. Dynamic ElementTextures.
ACM Trans. Graph.
32, 4, Article 90 (July 2013), 10 pages. https://doi.org/10.1145/2461912.2461921Chongyang Ma, Li-Yi Wei, and Xin Tong. 2011. Discrete Element Textures.
ACMTrans. Graph.
30, 4, Article 62 (July 2011), 10 pages. https://doi.org/10.1145/2010324.1964957Jonàs Martínez, Jérémie Dumas, Sylvain Lefebvre, and Li-Yi Wei. 2015. Structure andAppearance Optimization for Controllable Shape Design.
ACM Trans. Graph.
34, 6,Article 229 (Oct. 2015), 11 pages. https://doi.org/10.1145/2816795.2818101Paul Merrell and Dinesh Manocha. 2010. Example-based curve synthesis.
Computers &Graphics
34, 4 (2010), 304–311.Mathieu Nancel and Andy Cockburn. 2014. Causality: A Conceptual Model of Interac-tion History. In
CHI ’14 . 1777–1786. https://doi.org/10.1145/2556288.2556990Hans Pedersen and Karan Singh. 2006. Organic Labyrinths and Mazes. In
NPAR ’06 .79–86. https://doi.org/10.1145/1124728.1124742Mengqi Peng, Li-Yi Wei, Rubaiat Habib Kazi, and Vladimir G. Kim. 2020. AutocompleteAnimated Sculpting. In
UIST ’20 . https://doi.org/10.1145/3379337.3415884Mengqi Peng, Jun Xing, and Li-Yi Wei. 2018. Autocomplete 3D Sculpting.
ACM Trans.Graph.
37, 4, Article 132 (July 2018), 15 pages. https://doi.org/10.1145/3197517.3201297Kaspar Riesen and Horst Bunke. 2009. Approximate graph edit distance computationby means of bipartite graph matching.
Image and Vision computing
27, 7 (2009),950–959.Riccardo Roveri, A Cengiz Öztireli, Sebastian Martin, Barbara Solenthaler, and MarkusGross. 2015. Example based repetitive structure synthesis.
Computer Graphics Forum
34, 5 (2015), 39–52.Christian Santoni and Fabio Pellacini. 2016. gTangle: A Grammar for the ProceduralGeneration of Tangle Patterns.
ACM Trans. Graph.
35, 6, Article 182 (Nov. 2016),11 pages. https://doi.org/10.1145/2980179.2982417Christian Schumacher, Bernhard Thomaszewski, and Markus Gross. 2016. Stenciling:Designing Structurally-Sound Surfaces with Decorative Patterns.
Computer GraphicsForum
35, 5 (2016), 101–110.Ryo Suzuki, Tom Yeh, Koji Yatani, and Mark D Gross. 2017. Autocomplete Textures for3D Printing. arXiv preprint arXiv:1703.05700 (2017).Kenshi Takayama, Olga Sorkine, Andrew Nealen, and Takeo Igarashi. 2010. VolumetricModeling with Diffusion Surfaces. In
SIGGRAPH ASIA ’10 . Article Article 180, 8 pages.https://doi.org/10.1145/1866158.1866202Peihan Tu. 2020. Continuous Curve Textures Source Code. https://github.com/tph9608/continuous-curve-texture/.Peihan Tu, Dani Lischinski, and Hui Huang. 2019. Point Pattern Synthesis via IrregularConvolution.
Computer Graphics Forum
38, 5 (2019), 109–122. https://doi.org/10.1111/cgf.13793Lvdi Wang, Yizhou Yu, Kun Zhou, and Baining Guo. 2011. Multiscale vector volumes.
ACM Transactions on Graphics (TOG)
30, 6 (2011), 1–8.Lvdi Wang, Kun Zhou, Yizhou Yu, and Baining Guo. 2010. Vector solid textures.
ACMTransactions on Graphics (TOG)
29, 4 (2010), 1–8.Li-Yi Wei. 2016. Texture Synthesis. https://github.com/1iyiwei/textureLi-Yi Wei, Sylvain Lefebvre, Vivek Kwatra, and Greg Turk. 2009. State of the Artin Example-based Texture Synthesis. In
Eurographics 2009, State of the Art Report,EG-STAR
SIGGRAPH ’00 . 479–488. https://doi.org/10.1145/344779.345009Li-Yi Wei and Marc Levoy. 2001. Texture Synthesis over Arbitrary Manifold Surfaces.In
SIGGRAPH ’01 . 355–360. https://doi.org/10.1145/383259.383298Jun Xing, Hsiang-Ting Chen, and Li-Yi Wei. 2014. Autocomplete Painting Repetitions.
ACM Trans. Graph.
33, 6, Article 172 (Nov. 2014), 11 pages. https://doi.org/10.1145/2661229.2661247Jun Xing, Li-Yi Wei, Takaaki Shiratori, and Koji Yatani. 2015. Autocomplete Hand-drawn Animations.
ACM Trans. Graph.
34, 6, Article 169 (Oct. 2015), 11 pages.https://doi.org/10.1145/2816795.2818079Jonas Zehnder, Stelian Coros, and Bernhard Thomaszewski. 2016. DesigningStructurally-sound Ornamental Curve Networks.
ACM Trans. Graph.
35, 4, Ar-ticle 99 (July 2016), 10 pages. https://doi.org/10.1145/2897824.2925888Howard Zhou, Jie Sun, Greg Turk, and James M. Rehg. 2007. Terrain Synthesis fromDigital Elevation Models.
IEEE Transactions on Visualization and Computer Graphics
13, 4 (July 2007), 834–848. https://doi.org/10.1109/TVCG.2007.1027Kun Zhou, Xin Huang, Xi Wang, Yiying Tong, Mathieu Desbrun, Baining Guo, andHeung-Yeung Shum. 2006. Mesh Quilting for Geometric Texture Synthesis.
ACMTrans. Graph.
25, 3 (July 2006), 690–697. https://doi.org/10.1145/1141911.1141942Shizhe Zhou, Changyun Jiang, and Sylvain Lefebvre. 2014. Topology-constrainedSynthesis of Vector Patterns.
ACM Trans. Graph.
33, 6, Article 215 (Nov. 2014),11 pages. https://doi.org/10.1145/2661229.2661238C. Lawrence Zitnick. 2013. Handwriting Beautification Using Token Means.
ACM Trans.Graph.
32, 4, Article 53 (July 2013), 8 pages. https://doi.org/10.1145/2461912.2461985ACM Trans. Graph., Vol. 39, No. 6, Article 168. Publication date: December 2020. ontinuous Curve Textures • 168:15
A EXISTENCE ASSIGNMENT
The algorithm for generating additional samples is shown in Algo-rithm 1. function GenerateNewOutputSamples ( { n ( 𝑠 𝑖 ) , n ( 𝑠 𝑜 )} ) { 𝑠 𝑐𝑜 } ← ∅ {candidate sample set} {S} ← ∅ { S is a cluster that contains some candidate samples} { 𝑠 𝑜 } ← ∅ {new output sample set} for n ( 𝑠 𝑖 ) , n ( 𝑠 𝑜 ) ∈ { n ( 𝑠 𝑖 ) , n ( 𝑠 𝑜 )} do for 𝑠 ′ 𝑖 ∈ n ( 𝑠 𝑖 ) do if 𝑠 ′ 𝑖 is unmatched then Generate a candidate sample 𝑠 𝑐𝑜 with p ( 𝑠 𝑐𝑜 ) = p ( 𝑠 ′ 𝑖 ) − p ( 𝑠 𝑖 ) + p ( 𝑠 𝑜 ) and other attributes are the same to 𝑠 ′ 𝑖 { 𝑠 𝑐𝑜 } ← { 𝑠 𝑐𝑜 } ∪ 𝑠 𝑐𝑜 end if end for end for for 𝑠 𝑐𝑜 ∈ { 𝑠 𝑐𝑜 } do {Generate clusters from the candidate sample set by greed-ily looping over all candidates; more advanced clusteringtechnique can be applied to replace this step} Find the S within {S} with nearest center to 𝑠 𝑐𝑜 if Distance ( S , 𝑠 𝑐𝑜 ) < 0 . 𝛿 then { Distance ( S , 𝑠 𝑐𝑜 ) computes the spatial distance betweenthe center of S and 𝑠 𝑐𝑜 } S ← S ∪ 𝑠 𝑐𝑜 else S ′ ← ∅ S ′ ← S ′ ∪ 𝑠 𝑐𝑜 {S} ← {S} ∪ S ′ end if end for for S ∈ {S} do create a new 𝑠 𝑜 with averaged sample positions and attributesby merging all { 𝑠 𝑐𝑜 } within S 𝑖 ( 𝑠 𝑜 ) ← S n over 𝑠 𝑜 {existence assignment} if 𝑖 ( 𝑠 𝑜 ) > . then { 𝑠 𝑜 } ← 𝑠 𝑜 end if end for return { 𝑠 𝑜 } Algorithm 1.
Generating new output samples in existence assignment.
The candidate samples are generated from pairs of n ( 𝑠 𝑖 ) and n ( 𝑠 𝑜 ) (lines 5-12 in Algorithm 1). In a pair of n ( 𝑠 𝑖 ) and n ( 𝑠 𝑜 ) , if thereis an unmatched input sample 𝑠 ′ 𝑖 in n ( 𝑠 𝑖 ) (line 7), it will indicatethe potential lack of an output sample, whose global position is p ( 𝑠 𝑐𝑜 ) = p ( 𝑠 ′ 𝑖 ) − p ( 𝑠 𝑖 ) + p ( 𝑠 𝑜 ) located within n ( 𝑠 𝑜 ) , and attributesare the same to 𝑠 ′ 𝑖 (line 8). The algorithm loops over all pairs ofneighborhoods, each neighborhood pair may or may not producenew candidate samples. All these samples 𝑠 𝑐𝑜 form a candidate sampleset { 𝑠 𝑐𝑜 } . We group { 𝑠 𝑐𝑜 } into clusters {S} (line 13-24) by assigninga sample to its nearest cluster S with distance between 𝑠 𝑐𝑜 and thecenter of S smaller than 0 . 𝛿 or otherwise create a new cluster S ′ using the sample. For each cluster S ∈ {S} (line 25-31), wemerge all its candidate samples { 𝑠 𝑐𝑜 } ∈ S as one output sample 𝑠 𝑜 by averaging their position and attributes using the same way as inthe assignment step (Section 4.3.4). The existence of 𝑠 𝑜 is assignedas the ratio of the number of candidates within S over the numberof overlapping n over the position of 𝑠 𝑜 . For the sake of explanation,assume 𝑖 ( 𝑠 𝑜 ) =
1, it means all n overlapping over 𝑠 𝑜 produce onecandidate sample on average, which suggests there could be missingsamples, around 𝑠 𝑜 . Finally, 𝑠 𝑜 with 𝑖 > . B PARAMETERS
Table 2.
Parameters.
From left to right: neighborhood radii and samplingdistances from lower to higher hierarchies. The parameters in the bottompart of the table share default values. Input size is bounding box size ofinput exemplar. 𝑟 𝛿 input sizeFigure 12h { , , } { , , } × { , } { , } × { , , } { , , } × { , , } { , , } × { , , } { , , } × ... ... × × × × ... ... × × × { , , } { , , } × × × × ... ... × × × × × ... ... × × × × × C PERFORMANCE
Our current implementation in C++ is unoptimized. It takes about160 seconds to synthesize a pattern with about 750, 1000, 1300 out-put samples and 30, 30, 20 samples on average within neighborhoodsat each hierarchy, on a desktop with AMD Ryzen 9 3950 X 3.49 GHz16-core processor and 32 GB RAM. The major computational burdenis on the neighborhood searching process (Section 4.3.3). The com-putational complexity of neighborhood searching mainly depends
ACM Trans. Graph., Vol. 39, No. 6, Article 168. Publication date: December 2020. on the number of samples and neighborhood radius. To compute thesimilarity and sample matching between a pair of neighborhoods,the Hungarian algorithm [Kuhn 1955] has complexity 𝑂 ( 𝑁 n ) where 𝑁 n is the number of samples within a neighborhoood (assume all in-put and output neighborhoods have same number of samples). Thepatch match algorithm [Barnes et al. 2009] is composed of two al-ternating steps: propagation and random search. In an optimizationstep, there are the 𝑎𝑛 O 𝐼 𝑚𝑎𝑥 neighborhood matching computationwhere 𝑎 is a constant (in our implementation 𝑎 ≈
10) which is re-lated to the number of neighboring samples to a sample used in thepropagation step and the range of random search, 𝑛 O is the totalnumber of output samples, and 𝐼 𝑚𝑎𝑥 is the maximum number ofpatch match iteration. The complexity of an optimization step isthus 𝑂 ( 𝑛 O 𝑁 n ) . Our algorithm has no more than 3 hierarchies andeach hierarchy need about 7 steps. The lower hierarchy has lesssamples but a larger neighborhood radius (Section 4.3.5). D TEXTURE SYNTHESIS
Exemplar [Cornet andRouquier 2004] [Wei 2016] OursFig. 19.
Comparison of our algorithm to texture synthesis.
Texture synthesismethods need additional vector-pixel conversions and need to process allpixels instead of just samples around patterns.