FAME: 3D Shape Generation via Functionality-Aware Model Evolution
Yanran Guan, Han Liu, Kun Liu, Kangxue Yin, Ruizhen Hu, Oliver van Kaick, Yan Zhang, Ersin Yumer, Nathan Carr, Radomir Mech, Hao Zhang
11
3D Shape Generation viaFunctionality-Aware Model Evolution
Yanran Guan, Han Liu, Kun Liu, Kangxue Yin, Ruizhen Hu, Oliver van Kaick,Yan Zhang, Ersin Yumer, Nathan Carr, Radomir Mech, Hao Zhang
Abstract —We introduce a modeling tool which can evolve a set of 3D objects in a functionality-aware manner. Our goal is for theevolution to generate large and diverse sets of plausible 3D objects for data augmentation, constrained modeling, as well asopen-ended exploration to possibly inspire new designs. Starting with an initial population of 3D objects belonging to one or more functional categories, we evolve the shapes through part recombination to produce generations of hybrids or crossbreeds betweenparents from the heterogeneous shape collection. Evolutionary selection of offsprings is guided both by a functional plausibility score derived from functionality analysis of shapes in the initial population and user preference, as in a design gallery. Since cross-categoryhybridization may result in offsprings not belonging to any of the known functional categories, we develop a means for functionalitypartial matching to evaluate functional plausibility on partial shapes. We show a variety of plausible hybrid shapes generated by ourfunctionality-aware model evolution, which can complement existing datasets as training data and boost the performance ofcontemporary data-driven segmentation schemes, especially in challenging cases. Our tool supports constrained modeling, allowingusers to restrict or steer the model evolution with functionality labels. At the same time, unexpected yet functional object prototypes canemerge during open-ended exploration owing to structure breaking when evolving a heterogeneous collection.
Index Terms —Cross-category hybrids, functionality-aware shape modeling, functionality partial matching, set evolution. (cid:70)
NTRODUCTION D object modeling lies at the core of computer graphics.With the resurgence of VR/AR, deep learning, and3D printing technologies in recent years, there is a strongpush to develop innovative tools for 3D content creation,which can play key roles in data augmentation and thedesign and rapid prototyping of real 3D products. Currentmodeling tools have mainly focused on appearance, style,and aesthetic aspects of the generated shapes. One impor-tant criterion that is largely missing is functionality . Whencustomizing an existing design or evolving current designsinto a new prototype, the most basic requirement is for thefinal products to serve their intended functional purposes.In addition, rapid developments in geometric deep learningare placing an ever increasing demand for 3D models. Atool that is capable of producing functionally plausible 3Dshapes in large volumes and varieties is highly desirable.In this paper, we introduce a modeling tool which can evolve a set of 3D objects in a functionality-aware manner. Givenan initial population of 3D objects belonging to one or more functional categories, our tool produces generations and • Yanran Guan and Oliver van Kaick are with the School of ComputerScience, Carleton University, Canada. • Han Liu is with Electronic Arts, USA. • Kun Liu and Yan Zhang are with the Department of Computer Scienceand Technology, Nanjing University, China. • Ruizhen Hu is with the College of Computer Science and SoftwareEngineering, Shenzhen University, China. • Ersin Yumer is with Uber ATG, USA. • Nathan Carr and Radomir Mech are with Adobe Systems, USA. • Hao Zhang is with the School of Computing Science, Simon Fraser
University, Canada.
Fig. 1: Starting from a heterogeneous collection of four objects(left) as initial population, our functionality-aware evolution tool can generate a variety of plausible hybrid shapes. Twoof them (middle) exhibit strong functional similarity toexiting professional designs (right).generations of functionally plausible hybrids or crossbreedsbetween parents from preceding generations. For example,we could crossbreed a rocking crib and a chair into a combowhich can comfort both a parent and a baby; see Fig. 1(top). Taking an evolutionary approach to shape generationemphasizes that modeling is a continuous process of con-trolled diversification . Our main goal is for the evolutionarymodeling tool to produce many plausible 3D prototypes thatare both “ fit ” and “ diverse ” [31], so that they can augmentexisting datasets to boost learning algorithms and allowboth constrained modeling and open-ended exploration topossibly inspire new designs.The main challenges to functionality-aware model evolu-tion are two-fold. The first is attributed to cross-category modeling. Unlike all previous works on co-analysis anddata-driven shape processing [30], our tool works with a r X i v : . [ c s . G R ] M a y Input set Part group creation .........
Crossover with functionality partial matchingUnconstrainedConstrained sitting, rolling ... ... g g g g g Fig. 2: Starting from a set of segmented 3D shapes, we construct part groups for each shape, where a part group g i iscomposed of a combination of one or more shape parts (left). We apply crossover operations that replace a part groupin one shape by a part group of another shape, to create a variety of novel shapes (right). The evolutionary process canbe performed in an unconstrained or constrained manner. In the constrained evolution, the user prescribes functionalitiesthat should appear in the output shapes, e.g., sitting and rolling . The resulting shapes are then ranked according to theirfunctional plausibility. Finally, the user can select shapes to be part of the next generation for further evolution.a heterogeneous shape collection and composes parts fromdifferent object or functional categories. As a result, the well-known notion of structure-preserving shape modeling [20]can no longer be strictly adhered to; some levels of struc-ture breaking must occur. The second challenge is to adaptand enhance functionality models which were designedfor discriminative analysis [9], [11], [23] to serve shapemodeling. In particular, we must address the issue with newobject categories arising from cross-category hybrids whosefunctionality models could not be learned in advance.Starting with an initial population of segmented 3D objects,we first obtain a functional understanding of the input.Then we evolve the population, where at each iteration,the objects undergo stochastic part recombination, mimick-ing crossovers in evolutionary biology. Part deformationis applied to properly scale and connect the parts aftera crossover. Evolutionary selection of offsprings, i.e., thefitness criterion, is guided both by a functional plausibilityscore derived from functionality analysis of the initial pop-ulation and user preference, as in a design gallery [18],[31]. Specifically, users can express preferences over likedoffsprings, which will subsequently enter the population asparents while the remaining shapes would go extinct. Inaddition, our tool supports constrained modeling, allowingusers to restrict or steer the model evolution with constraintsdefined via intuitive functionality labels, such as sitting , leaning , and storage . For example, the user can pick twofunctionalities and constrain the model evolution to produceoffsprings possessing the two specified functionalities.Our evolution operates at the part group level, where a partgroup consists of one or more related shape parts. Evolutionis applied to a set (current population) of shapes, whereeach modeling operation is a crossover (i.e., an exchangeof part groups between two 3D shapes), or mutation (i.e.,part insertion into a 3D shape), followed by part (group)deformation and connection to improve plausibility of theoffspring. At each iteration, we generate models that satisfythe modeling constraints, rank them by their functional plausibility scores, and select the top k offsprings whilerespecting user-expressed preferences to enter the evolvingpopulation. Since cross-category hybridization may resultin offsprings not belonging to any of the known func-tional categories, we develop a means for functionality partialmatching by localizing the functionality models developedby Hu et al. [9] from the category or object level down topatch/part level. We then combine the functionality partialmatching scores against the known functional categoriesinto a plausibility score for the offspring; see Fig. 2.Our main contributions are summarized and reasoned, inthe context of the state-of-the-art, as follows: • We present the first 3D shape modeling tool basedon functionality-aware model evolution. By evolvingsets of 3D shapes progressively, our tool enables thegeneration of large volumes of functionally plausible and diverse hybrid objects for the first time, leading toapplications such as data augmentation, constrainedmodeling, and open-ended design exploration. • We generate new shapes via part recombination,possibly across object caragories. Importantly, ourmethod goes beyond exchanging parts sharing thesame functionality label, since merely having thefunctional parts in a shape does not imply that theresulting object would fulfill its intended function, asexemplified by the well-known “impossible teapot”design [21]. To this end, we develop a novel means ofevaluating the functionality of the assembled modelsin a global manner so as to enforce a proper, i.e.,functional, configuration of the combined parts. • To enable functionality evaluation of new hybirdswhich may not belong to any known object cate-gories, we introduce a key new concept, namely func-tionality partial matching , which analyzes functionalplausibility of new shapes not as whole, but in parts,with respect to learned functionality models. • Our interactive modeling tool requires only light userinteractions in the design gallery. It supports both open-ended object exploration and constrained mod-eling based on user-prescribed functionality labelsfor the target shapes, e.g., sitting and rolling .Typically, the initial population only consists of a handfulof 3D objects from one or more functional categories. Thecontinuous evolution is able to produce a large and diverseset of outcomes. We show a variety of plausible hybrids gen-erated by our functionality-aware evolution, demonstratingcontrolled design exploration via constrained modeling, aswell as the emergence of structure breaking and unexpectedyet functional object prototypes; see Fig. 1.An important utility offered by our new modeling tool isdata augmentation. We show that shapes generated by ourtool can complement existing datasets such as ShapeNet [1]and improve the diversity of training shapes and generaliz-ability of data-driven segmentation schemes, especially for atypical inputs such as shapes with missing parts.
ELATED W ORK
Classical shape modeling in computer graphics is subjectedto precise geometric constraints or controls, while fulfillinglow-level modeling criteria such as surface smoothness anddetail preservation [29]. Recently, much effort has beendevoted to structure-preserving shape processing [20], withmore emphasis on analyzing and manipulating part struc-tures that are characteristic of the class of processed 3Dobjects. With geometric modeling playing an increasing rolein the design and customization of 3D products, the researchand development focuses are naturally shifting to higher-level modeling criteria such as functionality.
Data-driven model assembly . In their earlier work on “mod-eling by example”, Funkhouser et al. [7] pioneered thedirection of data-driven 3D object modeling via search-and-assemble schemes applied to object parts [30]. Parts from 3Dobject exemplars can be substituted (into other objects) orrecombined to form new object prototypes [2], [3], [12], [15],[26], [31]. Mechanisms developed for determining suitablepart suggestions include geometric part similarity [7], [15],fuzzy correspondence [14], [31], and more often probabilisticreasoning learned from examples [2], [12].Our modeling tool is also data-driven and assembly-based,operating at the part group level. The key distinction andnovelty of our approach is the incorporation of function-ality analysis into the process. Moreover, all of the meth-ods mentioned above perform part compositions betweenshapes belonging to the same category, while our crossoveroperations can span different categories.
Evolutionary modeling and design . The seminal works of KarlSims [27], [28] introduced genetic algorithms to the graphicscommunity for the synthesis of novel creatures with desiredphysical behavior. Many follow-up works have appearedsince, e.g., the creature academy of Pilat and Jacob [22] andautomatic design and manufacturing of robotic forms [16].Most closely related, and inspiring, to our work is the “fitand diverse” modeling tool of Xu et al. [31], which alsoevolves a set of 3D shapes, via part mutation and crossover, while utilizing a design gallery [18] interface. Again, ourwork distinguishes itself from these previous works by con-sidering object functionality. “fit and diverse” only evolvesmodels of a same category, while our evolution is cross-category.
Structure preservation vs. breaking . Structure-preserving mod-eling and editing of man-made objects has attracted muchinterest in computer graphics [20]. This type of modelingis naturally performed only over shapes which belong tothe same category since it is the shared structures amongthese shapes that are to be discovered and preserved. Inour work however, we seek functionally plausible offspringswhich may not preserve prominent structures of their par-ents. Specifically, structure breaking is possible since ourfunctional plausibility score is not always positively corre-lated with structure preservation — it does not account forstructural constraints such as symmetry. As well, the com-position of part groups does not need to respect symmetry,e.g., one armrest of a chair can form its own part group.Our evolution is not designed to be strictly functionality-preserving either, as the hybrids can typically break someaspects of the functionalities of their parents.
Functionality analysis . Recently, there have been increasinginterests in studying 3D shapes from a functional perspec-tive [8]. Existing works on this topic have all focused oncharacterizing, comparing, or categorizing 3D objects basedon their functions, which are typically inferred from theirgeometry and interactions with other objects or agents [9],[10], [11], [13], [23], [25]. We are not aware of previous workson functionality-aware generation or synthesis of sets of 3Dobjects. Our work utilizes the category functionality modeldeveloped by Hu et al. [9]. However, the model has to belocalized to allow functionality partial matching, which isessential when defining the plausibility of hybrid shapeswhich do not belong to any known functional categories.Note that, as an application of their functionality model, Huet al. [9] were able to produce some limited forms of modelhybrids. However, the parent shapes are given and themodeling is manual, with functionality prediction resultsguiding the user in choosing where hybridization can occur.Most importantly, their hybridization is designed to preserve the functionality of the parent shape.
Cross-category part replacement . The modeling tool developedby Zheng et al. [32] does allow a specific form of 3D partsub-structures to be transplanted across objects categories.The key difference to our method is that their substructurereplacements still preserve the objects’ overall structures,i.e., there is no structure breaking. Also, the detected andreplaced substructures are rather specific: a symmetric ar-rangement of three parts called an S F ARR . In contrast, ourtool for cross-category modeling extracts functional prop-erties of 3D objects from a more generic analysis, and withstructure breaking, the models resulting from part crossoverexhibit much greater generality and versatility (Fig. 16).More closely related to our work is the modeling tool by Fuet al. [6] which generates part assemblies based on affordanceconstraints provided by a human pose. Specifically, theirmethod first identifies groups of candidate shapes which provide affordances compatible with the input pose, andthen recombines them, possibly crossing object categories,to form a coherent composite shape. In contrast, our func-tionality analysis is entirely based on shape geometry. Moreimportantly, our model evolution operates on 3D shapecollections and leads to the generation of a larger volumeand variety of functionally plausible hybrids, not just onemodel to fit a particular human pose, as demonstratedin Section 6. Finally, our modeling tool can also take onuser constraints, but in the form of functionality labels forthe target shapes; such constraints are more intuitve, morenatural, and much easier to specify from a user’s perspectivecompared to input human poses.
VOLUTION S ETUP
In this section, we describe the setup of the evolution,including the input and output of the method, and the pre-processing of the input.
Input and output . The input to our functionality-aware mod-eling tool, which serves as the initial population for theevolution, is a set of 3D shapes. Each shape comes with afine-grained segmentation into meaningful parts and con-tact points defined for each part. The input shapes are alsoroughly aligned in a consistent manner. The goal of theevolution is to stochastically create a large variety of shapesfrom a small set of parent shapes, according to guidancefrom functionality models and possibly the user. The inputsets that we use in our experiments typically have less than shapes. The additional work of segmenting the inputshapes is leveraged by the method for creating a large anddiverse set of offspring. Iteratively, we perform crossoveroperations on selected pairs of shapes from the populationto produce potential offspring. After several iterations, theoutput of the evolution is a population of shapes, whereeach shape combines parts coming from multiple parentshapes, possibly from distinct categories. Fig. 2 gives anoverview of the evolution, with more details in Section 4. Shape representation . The input shapes are given as trianglemeshes where the triangles are grouped into parts. For eachpair of adjacent parts, contact points indicate how the partsconnect to each other. As illustrated in Fig. 3, our methodconsiders three types of contact points, which connect twoparts with four, two, or a single contact point. These typesof contact points cover the majority of part connections ob-served in our dataset and allow us to appropriately positionparts by defining automatic connection rules. We extract thecontact points in a semi-automatic manner. For each pair ofparts, we detect the pair of closest points on each part andtake the midpoint of these points as the contact point. Then,if needed, the user can manually specify additional contactpoints or adjust them manually. In our dataset, about half ofthe contact points were computed automatically.Given all the contact points, a shape is abstracted as a relation graph , where each part corresponds to a node in thegraph, and two nodes are connected by an edge if the partspossess at least one contact point in common. Examples ofrelation graphs are shown in Fig. 4. The graphs capture the Fig. 3: Examples of part pairs with different numbers ofcontact points: four, two, and one contact point.Fig. 4: Examples of relation graphs for two shapes: a con-nected graph (left) and a disconnected graph (right).structure of the shapes and guide the placement of partsduring evolution.
Functionality labels . In our work, we employ two types of functionality labels : (i) part labels , which describe the function-ality of a part, e.g., rolling for a wheel or sitting for a chairseat; (ii) category labels , which denote the functionality of anentire category of shapes, e.g., chair for shapes that can beused as chairs. To avoid confusion between these two typesof labels, we call the former label of a part and the latter onlythe category of the shape.Before starting the evolution, we label the parts of eachshape with functionality labels. These serve as constraintsfor the evolutionary operations. To obtain the functionalitylabels for shapes of known categories, we first predict func-tional patches over the initial population of shapes with thefunctionality models of Hu et al. [9]. The prediction providesa weight field over the input point cloud, which defines theprobability that each point belongs to a patch that provides acertain functionality, such as sitting or leaning . These patchesassociated with a functionality are called proto-patches . Moredetails about the functionality models and prediction aregiven in Section 5.Given the weight fields predicted for each proto-patch, weassign to a part the label of the proto-patch that has thehighest sum of weights in the part. If the sum of weightsis below a threshold of . for all the functional categories(where the weights for a proto-patch sum up to ), thenwe leave the part unlabeled. If the input shape belongsto an unknown category, the user can manually annotateits parts with custom functionality labels, e.g., rolling . Notethat some parts may remain unlabeled. We keep track ofsymmetries among shape parts by storing them in a list. Inour work, symmetries are manually specified by the user,but any existing method for automatically detecting partsymmetries can be incorporated into our pipeline [19].Moreover, we assign each shape to a single functional cate-gory. Then, when shapes are evolved from two parents, theyreceive all the categories from their parents. The categoriesindicate which functionality models should be used for eval- : support: storage b b g g g g Fig. 5: Examples of part groups derived from base groups.Given base group b , with functionality label support , wederive part groups g and g by adding the colored partsto b , where b is formed by symmetric and disconnectedparts. Similarly, we derive g and g from b .uating the functionality of the shapes, which we describe inmore detail in Section 4. Part groups . Shape evolution is executed by crossover op-erations defined at the part group level, where each partgroup consists of one or more parts from a shape in thepopulation. We create multiple candidate part groups foreach shape by enumerating possible combinations of itsparts. We use a heuristic based on the part labels to formthe part groups, which helps us to avoid the combinatorialexplosion of enumerating all possible part groups.Specifically, we first group together adjacent parts withthe same functionality label, which form base groups fromwhich we build the final part groups. For each of these basegroups, we create several part groups by adding differentcombinations of parts that are adjacent to the base group,with a breadth-first search. Note that the expanded partsneed to be either all unlabeled or share a same label, whichcan be distinct from the initial label. A base group by itselfis also considered a valid part group. Part groups created inthis manner are restricted to one or two functionality labels,and thus form meaningful structures for exchanging andrecombining the functionalities of the input shapes. We alsoallow symmetric parts to form base groups for further enu-meration, implying that part groups can be disconnected.Finally, individual symmetric parts can also form the basisof part groups, so that structure breaking is allowed in theevolution. Fig. 5 shows examples of part groups created fortwo different types of base groups of a shape.
HAPE E VOLUTION
We now describe our shape evolution framework, dis-cussing first the evolutionary operations and then describ-ing our method for constrained evolution. Scoring mecha-nisms which control the flow of the evolution will be thefocus of Section 5.
Evolutionary operations . A crossover is defined between twopart groups g A and g B , anchored on shapes S A and S B , re- Initial alignment Refined alignment Initial alignment Refined alignment g A g A g B g B Fig. 6: Two examples of part group exchange. After acrossover that replaces the part group g B by g A , we performan initial placement of g A based on bounding box align-ment. This is followed by a refinement deformation thataligns contact points (blue and orange circles). The refine-ment may be reverted back if some of the contact points arenot brought into proximity (right column example).spectively. The crossover results in two possible offsprings.In one offspring, g A is replaced by g B on shape S A . In theother, g B is replaced by g A on shape S B . We also allowin some cases g A (or g B ) to be the null set, so that oneof the offsprings would be S B with g B deleted and theother is the result of inserting g B into S A . We call thesetwo different types of crossover part group exchange and part group insertion . Deformation of part groups may benecessitated after a crossover to fulfill geometric constraints.Note that we do not define a part group removal operationsince we start the evolution with relatively simple shapesthat we evolve into more complex ones, so that removal offunctionality is not needed. Fig. 6 illustrates examples ofpart group exchange, while Fig. 7 illustrates insertion. Part group exchange . Suppose without loss of generalitythat we are replacing g B with g A in shape S B . We firstperform an initial alignment of g A , so that we can use spatialproximity to establish a correspondence between the contactpoints of g A and contact points that previously connected toparts in g B . Then, we refine the placement of g A so thatcorresponding contact points are brought into contact witheach other, although we do not explicitly merge the trianglemeshes that represent the parts.For the initial alignment, we describe g A and g B with axis-aligned bounding boxes. We translate g A so that the centerof its box aligns with the center of g B . Next, we scale thelongest axis of g A to align it with the corresponding axis of g B , and scale the other axes proportionally to maintain theaspect ratio of g A ’s bounding box.For the refined alignment, we match each contact point in g A to the closest contact point in S B , after the initial alignment.If g A and g B have different numbers of contact points n A and n B , respectively, we define the match only for the n = min( n A , n B ) closest contact points. Then, we derivethe transformation that best aligns the matching contactpoints. In the following operations, only points that werematched are considered. First we derive a translation thataligns the centroid of the contact points in g A to the centroid of the corresponding contact points in shape S B . Next, wescale g A by the average of the scalings needed to align eachcontact point in g A to its matching contact point in S B . Thisprovides a transformation that best aligns the part groupsin a least-squares sense, according to a term which is thesum of squared errors of all contact points, similarly to theoptimization proposed for part placement by Kalogerakis etal. [12]. Note that, since the input shapes are roughly pre-aligned, the transformation does not include a rotation.The two steps used in part placement are illustrated in thebottom row of Fig. 6. If the refined alignment is suboptimal,meaning that the distance of any of the contact points toits closest contact point in the other part group is too large,according to a threshold, then we revert back to the initialalignment. This provides a more meaningful part placementas shown in the right column of Fig. 6, since in this examplethe refined alignment fails because some of the blue contactpoints are too far from the yellow ones. The proximitythreshold is set as of the shape’s bounding box diagonal.During the refined alignment, the proportions of parts canchange, which could leave some parts unrecognizable. Thus,we restore the proportions of parts that possess functionalitylabels, if their scaling passes a threshold. Specifically, if thescaling of the y -axis or z -axis of a part’s bounding boxrelative to the x -axis is greater than an empirical factor of , we restore the scale of the affected axis. Part group insertion . For inserting a part group g into a shape S A , we look for a region in S A with a similar structure to thecontext of g in its source shape and place g in this region.Specifically, we record the relative position (translation vec-tors) of g to all of its adjacent parts in its source shape thathave functionality labels, where we denote the set of labelsof the adjacent parts as L adjacent . Next, we locate a regionin S A which has a similar relative position to the same setof labels L adjacent . This is formally defined as the locationin S A with the average of translation vectors that is theclosest to the average recorded in the source shape. Only thelabels in L adjacent that exist in shape S A are considered inthe average. We choose this location as the position to insert g . An example of insertion is shown in Fig. 7. Note that, ifthe location in S A is already occupied by a part group g A ,then we perform a standard crossover to exchange g A for g . Shape evolution . Shape evolution is performed with crossoveroperations and is guided by user preferences, including pre-ferred functionality labels and preferred offspring shapes.Starting from an initial population G , the evolution iter-atively generates offspring shapes in generation G i fromshapes in the previous generation G i − . All the generatedshapes preserve the user-selected functionality labels spec-ified by a set L user . The evolution stops when a presetmaximum number of iterations i max is reached.During the generation of G i , the evolution considers allpossible pairs of shapes, S A and S B ( S A (cid:54) = S B ), from G i − as parent shapes to generate offspring shapes. If all theuser-selected functionality labels are already present in theparent shapes, we only perform crossovers that exchangepart groups. If missing functionality labels are detected, weinsert a part group that possesses the missing functionality (a) Source part group (b) Similar structure (c) Insertion result Fig. 7: Two examples of part group insertion. Given a partgroup in (a), we find a region in the target shape in (b) witha structure similar to the context of the part group in thesource shape in (a). For example, the context consists of foursupport structures (the legs) adjacent to the basket on thetop row, or one support structure (one leg) adjacent to eachwheel on the bottom row. We insert the part group in thisregion as shown in (c).into the candidate shape. Then, we apply a diversity selec-tion to ensure that the generated shapes are geometrically diverse . Also, we use functionality partial matching, which isdiscussed in Section 5, to compute the functional plausibilityscores for each offspring shape, and sort the shapes inthe current generation according to the descending orderof their scores. Finally, after a generation G i is produced,we allow the user to select preferred offspring shapes asparent shapes for the next generation. The flow of our shapeevolution is described in Algorithm 1. More details aboutthe constraints used in the evolution are discussed below. User-in-the-loop modeling . The user can control the evolutionwith two guiding mechanisms: setting constraints on thefunctionality part labels, and selecting preferred offspringsfor further evolution. For constrained modeling, the user se-lects a set of part functionalities L user that should appear inthe offspring shapes, and the evolutionary process ensuresthat the shapes generated by crossovers possess all of thespecified labels. In practice, the user selects the labels froma list of the labels that appear in the input population.For satisfying the user constraints, we verify if all the shapespossess parts with the functionality labels listed in theconstraints. If a shape S A does not possess all the labels, weinsert the missing labels through additional crossover oper-ations. Specifically, for each missing functionality in S A , werandomly select a part group g that possesses the missingfunctionality, from the pool of all part groups derived fromthe parent shapes. Next, we add the part group to S A withone of two operations: (i) exchanging g for a part group g A which is unlabeled or has a label that is not in the set ofconstraints; (ii) inserting g into S A with part group insertion.We remark that insertion operations are mainly used to add ALGORITHM 1:
Shape evolution
Input: G , L user , i max (refer to the text for symbol notation) Output: G evolved function ShapeEvolution( G , L user , i max ) G evolved ← ∅ i ← while i ≤ i max do G i ← ∅ foreach S A ∈ G i − do L A ← functionality labels in S A L missing ← L user \ L A foreach S B ∈ G i − doif S A (cid:54) = S B thenif L missing = ∅ thenforeach g A ∈ part groups of S A doforeach g B ∈ part groups of S B doif g A is unlabeled or has nofunctionality label in L user then S offspring ← exchange g A on S A for g B G i ← G i ∪ {S offspring } if L missing (cid:54) = ∅ thenforeach g ∈ part groups of S B doif g has a functionality label in L missing then S offspring ← insert g into S A G i ← G i ∪ {S offspring }G i ← DiversitySelection( G i )scores ← ∅ foreach S ∈ G i do s ← FunctionalityPartialMatching( S )scores ← scores ∪ { s }G i ← Sort( G i ,scores) G i ← UserSelection( G i ) G evolved ← G evolved ∪ G i i ← i + 1 return G evolved missing functionality to offsprings, as otherwise they couldbe applied ad infinitum in an unconstrained setting. If thenecessary part group insertions cannot be performed, thenthe candidate shape is not retained.After the set of evolved shapes had been created, it canbe further filtered with additional validity checks. In ourimplementation, we include a diversity selection step. Diversity selection . Given that several of the offspring shapescan be geometrically similar if they were created fromsimilar part groups, we present to the user selected shapesthat are diverse in terms of their geometry. Specifically,we compute the geometric similarity between all pairs ofshapes according to the light field descriptor (LFD) [4],which gives an indication of the global similarity of shapes.Next, we perform farthest point sampling according to theLFD distances, to keep only the top
50 % most distinctshapes. These shapes are presented to the user accordingto a ranking that takes into account their plausibility from afunctional perspective, as described in the next section.
UNCTIONALITY A NALYSIS AND S CORING
In this section, we explain how we evaluate the functionalplausibility of the evolved shapes.
Category functionality model . We base our functionality anal-ysis on the model of Hu et al. [9]. We provide a shortdescription of this model here for completeness, but refer thereader to the original paper for more details [9]. Each modelcaptures the functionality of an entire category of shapeswith a set of proto-patches and their geometric descriptors.A proto-patch is a generalization of patches that support aspecific type of interaction, e.g., handles for grasping. Themodel stores the proto-patches for all the interactions thatare needed to enable the functionality of a category, e.g.,proto-patches of handles, baskets, and wheels of supermar-ket carts. Each shape is described as a set of points, and eachproto-patch is represented as a weight field defined over thepoints. The weight of a point represents the likelihood thatthe point belongs to the specific proto-patch.Besides the proto-patches, the complete model stores thetypical values of geometric descriptors of points that appearon the proto-patches, and also binary descriptors definedbetween pairs of points belonging to two different proto-patches. The proto-patches in the functionality models allowus to localize the functionality of the shapes. We use thislocalization to infer the functionality labels of parts, asdescribed in Section 3.The models are learned from a set of training shapes givenin contexts that describe the functionality of the shapes [9].Given an unknown shape, we can use the model of acategory to predict proto-patches on the shape, and thencompare the descriptor values of the proto-patches to thosestored in the model, to determine how well the shape sat-isfies the functionality of the category (its functionality score for the category). Note that we use functionality models for categories of shapes since these can be learned from existingdata [9], while generic models of functionality typicallydepend on ad hoc geometric rules that do not generalize.
Functionality partial matching . During the shape evolution,we can use the category models to compute the functionalityscores of the offsprings, to obtain an indication of their func-tional plausibility. However, given that the parent shapesare of different categories, we may generate cross-categoryoffsprings that do not fit well a single category. Moreover,one of the goals of the evolution is to generate shapes withsurprising functionality, possibly mixing the functionalityof different categories. Thus, we need to take multiplecategories into account when evaluating an offspring.We stipulate that a cross-category shape is functionallyplausible as long as it partially supports the functionality ofmultiple categories. Thus, we arrive at the notion of function-ality partial matching , where we derive the functional scoreof a shape by aggregating the scores of partial matchings ofparts of the shape to different category functionality models.Computing such partial matching requires addressing threechallenges. First, the functionality scores of partial shapesderived from the input shape need to be computed effi-ciently for a given category model. Second, the functionality ... ...
Fig. 8: Beam search for functionality partial matching, wherewe search for the subset of parts that provides the highestscore for a functional category ( chair in this example). Westart with the full shape and enumerate its subsets byremoving parts one at a time. We only expand a node if it ispromising, i.e., it has one of the top two functionality scores,as the beam width is . The node from level highlightedin red is returned as the best partial match, since its children(omitted from the figure) have lower scores. The full searchtree is shown in the supplementary material.scores of different categories need to be normalized, so thattheir scores can be mutually compared. Third, the scores ofmultiple categories need to be aggregated in a meaningfulmanner to provide the overall partial matching score. Wediscuss how we address these three challenges as follows. Functionality score of partial shapes . Given a category model M and a shape S to be evaluated, we enumerate subsetsof shape parts, including subsets of size , , and so on.Next, we consider each subset as an individual shape andevaluate its functionality according to M . Given that weare interested in the partial shape that best supports thefunctionality of the category, we search for the subset ofparts with highest functionality score. To avoid enumeratingand evaluating all possible subsets of parts, we perform theenumeration more efficiently with a beam search approachdescribed as follows.Since we experimentally observed that the full objects havethe highest functionality scores in a large number of in-stances, we perform a reverse beam search , where we startwith the entire shape and evaluate subsets obtained byremoving parts of the shape one at a time. In the search,we keep track of the w subsets of parts with the top func-tionality scores computed up to the current iteration, andcontinue expanding recursively only these subsets, where w is the beam width. We use w = 2 in our experiments,but always expand all of the nodes in the first level of thesearch. To expand a node in the search, we generate eachsubset obtained by removing one part from the node’s set ofparts. The search stops when the w nodes with higher scoreswere expanded, or when no further parts can be removedfrom any of the w nodes. Fig. 8 illustrates the beam search.In addition to having a high score, a partial shape createdduring the search should satisfy three constraints:(i) All parts of the partial shape should be connected. Sincewe use a relation graph to represent each shape, the part-wise connectivity check can be reduced to checking the sitting storage rolling Fig. 9: Subsets of parts with high partial matching score(shown in orange) which are functionally implausible dueto their surrounding space being obstructed by other parts.graph connectivity. For example, in Fig. 4, the relation graphon the left represents a shape with all parts connected, whilethe one on the right represents a disconnected shape.(ii) The partial shape should be physically stable. Similarto the idea of static stability of shapes introduced by Fuet al. [5], we verify whether the center of mass of thepartial shape falls within the polygon formed by all theground-touching points of the shape. In more detail, we firstcalculate the center of mass of the shape by averaging thecenter points of the bounding box diagonals of all parts ofthe shape. We also compute the convex polygon formed byall ground-touching points of the shape. To decide whetherpoints are touching the ground, we find the minimum z -coordinate z min among all points in the shape and mark allpoints that have z -coordinate smaller than z min + 1 % of thebounding box diagonal as ground-touching points. Then,the shape is considered as physically stable if the projectionof the center of mass on the ground is inside the convexpolygon. During the search, partial shapes are allowed to beunstable, since this may only be temporary while the finalnodes of the search will be stable. Thus, we keep the topstable shapes in addition to the top w partial shapes, in casenone of the top scoring shapes is stable.(iii) The partial shape should have an adequate functionalspace to support a function, e.g., a human may not sit ona seat if the seat is obstructed by other parts. We use thefunctional space extracted by the functionality model of Huet al. [9] to verify that the space needed to perform thefunction associated to the label is free. A few examples areshown in Fig. 9. Functionality score normalization . The functionality scorescomputed with the model of Hu et al. [9] are based onthe similarity between the input shape being evaluated andtraining shapes in the category of the model. Since thefunctionality models are trained separately per category,scores for different categories are not directly comparable.To address this limitation and obtain scores that are com-parable across categories, we normalize the scores for eachcategory according to the training data.Given a category model, we compute the scores for all theshapes in the training data. Next, we compute the cumula-tive distribution of the scores of shapes inside the category,and also of the scores of shapes outside the category, ob-taining cumulative distributions D and D , respectively.Given an unknown query shape, we compute its score withthe category model, and compute its normalized score as w × p + w × p , where p is the probability of the Shelf: 0.943 0.854 Chair: 0.947 0.786 p p p p Fig. 10: Normalization of functionality scores for the shapeon the left. The two numbers on the top of each graph arethe scores of the shape for the shelf and chair categoriesbefore and after normalization. The score distributions fortraining shapes inside/outside the corresponding categoryare drawn in green and red, respectively. The probabilities p and p represent the percentages of shapes having lowerscores in the corresponding distributions. We see that, eventhough the original scores for the two categories are quiteclose, the normalized score for the chair category becomesmuch smaller than that of the shelf category, since shapesinside the chair category have relatively high scores.shape’s score according to D , and p according to D .The probability p is an approximation of the percentageof shapes inside the category that have scores lower thanthe query, and represents the probability of the query beinginside the category. Similarly, p represents the probabilityof the shape not being outside the category. Thus, the queryis more likely to belong to the category when both p and p are large, which can be captured by a weightedsum of these two probabilities. The weights w and w are defined as the percentages of shapes inside and outsidethe category over all the shapes in the training data. Theyindicate the reliability of the two distributions, respectively.Fig. 10 shows one example to illustrate how the scores arenormalized and more meaningful than the original scores. Functional plausibility score . Given the best functionalityscores of a shape computed for each of the relevant categorymodels through partial matching and normalized as de-scribed above, we integrate the scores into a single numberthat indicates the shape’s cross-category functional plausibility by simply taking the maximum of the scores. Althoughdifferent manners of integrating the scores are possible, themaximum score indicates the best partial functionality thatthe shape possesses, and thus sets a lower bound for thefunctionality of the shape. An alternative approach suchas the sum of scores would be biased by the categoriesthat possess low scores because the shape does not supporttheir functionality. However, such low-score categories donot necessarily indicate that the shape is not functional.One limitation of the maximum is that it does not indicatewhether a shape has multiple functionalities. To take intoaccount such multi-functional shapes, the score describednext can be used as an alternative to the plausibility score.
Multi-functionality score . The goal of the multi-functionalityscore is to detect shapes that combine multiple, knownfunctionalities in unusual manners. This score can help inmeasuring the “surprise factor” or unpredictability of anoffspring. We define the score as the number of differentfunctionality categories that are partially supported by theshape. Specifically, we count the number of functionalitycategories for which the shape has a high score, determined by whether the maximum of the functionality partial match-ing probability for the category is above a threshold θ . Weuse θ = 0 . in all of our experiments. In Section 6, we showhow the multi-functionality score can be used in place of theplausibility score to rank shapes, when the goal is to selecthybrid shapes that support multiple functionalities. Trade-off between user and machine time . The two scores de-scribed above provide a good indication of the functionali-ties that a shape can support. In general, the evolved shapeswith high scores are plausible and thus the users can focuson selecting shapes based on their design interests. On theother hand, the computation of the scores via the functional-ity partial matching relies on an expensive search procedure,implying that the time to evolve a single generation can beconsiderably long. Thus, we also provide the user with anoption to run a simplified partial matching that helps tobalance between user and machine time. For each evolvedshape, we compute its functionality scores based only onthree sets of parts: the entire shape, and each part groupcoming from its two parents. This simplification speeds upthe computation time considerably, at the expense that theuser may have to inspect a larger number of shapes, sincesome plausible shapes may not be ranked correctly.
ESULTS , E
VALUATION , AND A PPLICATIONS
In this section, we show results of our functionality-awaremodel evolution under different modeling scenarios andalso evaluate various aspects of our modeling framework.We also provide comparisons to the modeling tools byZheng et al. [32] and Fu et al. [6]. Finally, we show anapplication of our modeling results to data augmentationfor shape segmentation.
Dataset and functionality models . Our experiments have beenconducted mainly on 3D objects belonging to functionalcategories including chairs, handcarts, baskets, etc. How-ever, as we shall show, the initial population of our modelevolution is not constrained to only include objects withknown functional categories. The functional categorieshave been adopted from the work of Hu et al. [9], which de-veloped category functionality models for these classes. Inour work, we leverage these functionality models with ourfunctionality partial matching so that they can be appliedto cross-category models. Note that a few categories suchas hook, hanger and backpack do not have many parts thatcan be exchanged. Thus, they do not appear in our results,as they provide only a small number of part groups. Parameters and statistics . All our experiments were conductedwith the same parameter settings, as described in preced-ing sections. As an example of the execution time, pre-processing of an initial population of shapes takes onaverage
15 min , including functional proto-patch mappingand part group creation. Then, evolving one generation withrandom crossovers from this population of shapes with apool of part groups (including null part groups) and functionality labels generates shapes. On the other hand,with user constraints (preserving two functionality labels),only offspring shapes are produced. For the execution G1G2G3Initial population
Fig. 11: A gallery of modeling results from unconstrained evolution obtained with our functionality-aware approach. Thefirst row shows two initial populations of four objects each, one per column. The next three rows show selected offspringfrom subsequent generations. The generations contain , , and shapes in total for the set in the first column, and , , and shapes for the set in the second column.time, if a simplified partial matching is adopted, the timeneeded to evolve one generation is about on average,where evolutionary operations take about (
17 % ofthe computation time for a generation) and functionalitypartial matching takes (the remaining
83 % of thecomputation). The time needed to compute the functionalplausibility score of a single offspring is
50 s on average withthe simplified partial matching. If a reverse beam search isemployed, the time for evolving one generation from thesame population is about
55 min , where evolutionary oper-ations take about ( of the computation time) andfunctionality partial matching takes
54 min (the remaining
98 % of the computation). The time needed to compute thefunctional plausibility score of a single offspring with thereverse beam search is on average .The largest population we evolved includes shapes,taking on average . to evolve one generation (with userconstraints and the simplified partial matching). All timingswere measured on a workstation with an Intel Core i7-6700 .
40 GHz quad-core processor. Our implementation isbased on C In this section, we present a variety of shape evolution re-sults obtained with our method. If not indicated otherwise,for each figure, we show selected shapes from the evolvedpopulations, mimicking the process where users wouldhandpick interesting offspring from a large and diverse set
1. Our implementation is available at the following URL: https://github.com/IsaacGuan/FAME. of shapes. In Fig. 14 and Fig. 15, we evaluate specifically therankings produced by our method.
Unconstrained evolution . Fig. 11 shows evolution results cor-responding to an open-ended exploration, where the useronly selected an initial population encompassing multi-ple functional categories, without adding additional con-straints. We can observe that many of the evolved shapes aretruly cross-category, combining the functionality of two ormore parents, such as the movable shelves and table-shelvesin the first column, or the sittable shelves in the secondcolumn. As the evolution advances, the complexity of theoffsprings also increases, where we see a variety of multi-functional objects. In the second column, we see in the initialpopulation, an object, the horse-shaped toy, that is not partof any of the pre-defined functional categories. By manuallylabeling the toy’s parts with appropriate functionality labelssuch as rocking and sitting , our evolution can combine theobject with other shapes to create offsprings that preservethese functionalities. Finally, we note that the functionalityof the parent shapes is preserved in the offsprings, withoutobstructions to the functional spaces.
Constrained modeling . Evolution guided by user constraintsis the modeling scenario where our tool is most effective.The user can start with a design goal to “generate shapesthat enable storage and transportation”, and select appro-priate functionality labels as constraints to ensure that thesefunctionalities appear in the offsprings. During evolution,our tool provides concrete examples of shapes satisfying theconstraints. The user can examine the generated shapes toselect and further evolve the preferred ones.Fig. 12 shows a first set of results of constrained evolution,where we show three generations that were evolved from G1: placement, graspingG2: evolving only selected shapes (blue color) from G1G3: storage, rollingInitial population
Fig. 12: Results of constrained evolution by ourfunctionality-aware modeling tool. The user evolves theinitial population by constraining the offsprings with thefunctionality labels placement and grasping , obtaining thefirst generation G1. The user then selects the shapes markedin blue in G1 to be further evolved, to get G2. Finally, thefunctionality constraints storage and rolling are included asnew preferences into the evolution of all the shapes, toobtain the third generation G3.one input population. We observe how the shapes of ageneration possess all of the functionalities specified asconstraints, while at the same time they provide differentvariations on how these functionalities can be enabled,e.g., several objects that combine placement of objects (likeshelves) with grasping of handles for transportation in G1and G2, and a variety of storage furniture like baskets andshelves that can be rolled in G3, where several of the evolvedobjects also inherit the grasping or placement aspects fromparents in G2.
Structure breaking . The ability to break structures, such assymmetries, during part composition is a unique feature ofour cross-category modeling tool, which allows the evolu-tion to introduce variations in the structure of the generatedshapes. Both results in Fig. 1 exhibit structure breaking. The sitting, leaning, grasping, placement, rollingplacement, grasping, rollingsitting, leaning, placement storage, graspingplacement, storage(a) Shapes with 2 functionalities(b) Shapes with 3 functionalities(c) Shapes with up to 5 functionalitiesplacement, grasping, storage, rolling
Fig. 13: Results of model evolution where the objects areconstrained to possess , , and up to functionalities.percentage of offspring exhibiting symmetry breaking canvary considerably, e.g.,
45 % for the set shown on the left ofFig. 11, and . for the set on the right. Scalability . We evaluate the scalability of our evolution intwo aspects: the number of shapes and number of con-straints involved. In Fig. 13, we see that the evolutionproduces plausible results when requesting a small numberof functionalities such as or , but also a larger numbersuch as . The generated shapes are true hybrids that canserve multiple functionalities, such as the chairs in Fig. 13(c)that, besides enabling sitting and leaning, also enable theplacement of objects, and rolling and grasping for trans-portation.Fig. 14 presents three sets obtained by evolving the sameinitial population of shapes by choosing different setsof functionality constraints. The input population is largerthan the input sets used in the previous experiments toassess scalability. We observe how the evolution generateslarge sets with hundreds of novel shapes which are allconstrained by the user guidance. Ranking scores . When a large number of offsprings are gen-erated, a ranking measure is especially important so thatthe most plausible objects are first presented to the user. InFig. 15, we compare the functional plausibility and multi-functionality measures as choices for ranking the shapesfrom Fig. 14. We observe how objects with high values forthe two measures appear more functional than objects withlow scores. In addition, objects with high multi-functionalityscores tend to combine several functionalities. Thus, thesemeasures enable the user to save time by only inspecting themost promising prototypes. We further evaluate the ranking sitting, leaningplacement, storagerolling, graspingInitial population Fig. 14: Results of model evolution from a large initial popu-lation ( shapes) and with various functionality constraintsto demonstrate scalability. The following populations aregenerated: sitting + leaning with shapes, placement + storage with shapes, and rolling + grasping with shapes. Only the top shapes for each set are shown,according to the ranking by functional plausibility.measures with two user studies that we present in the sup-plementary material, where we show that our functionalityscores are able to capture the multiple functionalities of ashape as perceived by humans. Comparison to Zheng et al. [32] . In Fig. 16, we compareour method to that of Zheng et al. [32] by applying ourmodel evolution to one of their input sets, with the sameshape segmentations. Recall that their method is designedto preserve a specific type of three-part symmetric supportsubstructure ( S F ARR ) in the input and limited to six combi-nation rules, while our method is more generic and evenallows symmetry breaking. We observe that our generichybridization/crossover approach, i.e., without explicitly (a) Low validity, low multi-func (b) High validity, low multi-func(c) High validity, high multi-func
Fig. 15: Comparison of ranking scores. Three sets of objectswith different levels of priority for plausibility and multi-functionality measures.
Input set from Zheng et al. [3 ]A selection from our resultsResults of Zheng et al. [3 ] Fig. 16: A comparison of our shape generation results tothose from Zheng et al. [32], on an input set from their work.The set of offsprings generated by our method contains notonly shapes producible by their method (shapes in yellow),but also other shapes (in blue) which their method cannotproduce for various reasons discussed in the text.modeling or enforcing any specific symmetries or supportstructures, can also generate the types of shapes that theirmethod can. At the same time, our modeling tool is ableto achieve more general part recombinations and produceshapes exhibiting larger functional variations, e.g., the com-bination of sitting and storage that appears in the couchhybridized with a bench and shelf, where two differentparts are attached to the supporting structure. Moreover,we remark that their method is able to generate interestingvariations for this input set where all shapes have symmetricsupport structures, but their method is not applicable tomore general shapes like the asymmetric desks that weevolved in Fig. 1 or Fig. 14.
Comparison to Fu et al. [6] . In Fig. 17, we compare ourmethod to that of Fu et al. [6], which can synthesize across-category, functional hybrid to fulfill the affordanceconstraint defined by an input human pose. The comparisonresults demonstrate that our evolutionary modeling tool (a) Input objects (b) Fu et al. [ ] (c) Our results Fig. 17: Comparison to functional hybrid generation by Fuet al. [6]. In each row, we show the 3D shapes identified bytheir method (left) that match a human pose and the hybridshape produced (middle). Using the same 3D shapes, ourmethod is able to generate a more diverse set of hybrids(right), including one that well resembles the outcome fromtheir method, without a human pose as constraint. Ϭ Ϯ Ϭ ϰ Ϭ ϲ Ϭ ϴ Ϭ ϭ Ϭ Ϭ ϭ Ϯ Ϭ ϭ ϰ Ϭ ϴ Ϯ ϴ ϰ ϴ ϲ ϴ ϴ ϵ Ϭ ϵ Ϯ Đ Đ Ƶ ƌ Ă Đ LJ ; й Ϳ Ś Ă ŝ ƌ Ɛ Ϭ Ϯ Ϭ ϰ Ϭ ϲ Ϭ ϴ Ϭ ϭ Ϭ Ϭ E Ƶ ŵ ď Ğ ƌ Ž Ĩ Ś LJ ď ƌ ŝ Ě Ɛ Ś Ă Ɖ Ğ Ɛ Ă Ě Ě Ğ Ě ƚ Ž ƚ ƌ Ă ŝ Ŷ ŝ Ŷ Ő Ɛ Ğ ƚ ϵ ϯ ϵ ϰ ϵ ϱ Đ Đ Ƶ ƌ Ă Đ LJ ; й Ϳ d Ă ď ů Ğ Ɛ Fig. 18: Results showing improved accuracy via data aug-mentation for learning shape segmentation, using PointNet,for two sets of shapes. The ShapeNet training set is aug-mented progressively with shapes evolved using our tool.Please refer to the text for details.is able to produce similar hybrids, without specifying ahuman pose or explicit affordance constraints, as well asother hybrids which are also functionally plausible. On theother hand, an input human pose does narrow down thesearch for potential parent shapes and part placements; theresulting synthesized shape would more closely serve aspecific target functionality. (a) ShapeNet set (b) The augmented set (c) Ground truth
Fig. 19: Visual results of PointNet segmentation on partialtest shapes, using ShapeNet training vs. the augmentedtraining set (ShapeNet + our shapes).
We have shown that our functionality-aware model evolu-tion is able to generate a large and diverse population offunctionally plausible offsprings. But, from the perspectiveof modeling and design exploration, we should expect thefunctionality requirement placed on the final outcomes tobe quite stringent. Hence, in an evolved population, thereare usually only few shapes that are fully functional or caninspire new designs. On the other hand, data augmentation,which is aimed at boosting the performance of learning-based shape analysis schemes, is a venue where we can uti-lize most, if not all, of the evolved shapes from our modelingtool. We expect these shapes to serve as useful training datasince they are both “fit”, i.e., plausible, to resemble, at leastin part, potential test shapes, and diverse, to provide a bettercoverage of the distribution of test shapes.To demonstrate the potential of our method for data aug-mentation, we use the generated offspring to augment train-ing data for one key application: shape segmentation of par-tial shapes. To create a set of labeled shape segmentations,we assign labels to the already segmented parent shapes,which usually constitute a small set that can be manuallylabeled by a human in a short time. Then, we keep the labelsassigned to each part as we evolve the parents into a largeset of segmented and labeled shapes. We use this data toaugment the training data for shape segmentation.Specifically, we evolve two input populations. The firstpopulation is composed of chairs and shapes fromother classes that add diversity to the set, resulting in anevolved population of hybrid chairs, since we constrainthe functionality of chairs to be present in the offspring.The second population is composed of tables and shapes from other classes, resulting in hybrid tables.For the experiments, we use PointNet [24] for segmentingpoint clouds, where we uniformly sample our shapes with points. To show how our data can aid in learning tosegment diverse shapes, especially when partial matchingis needed, we create a test set of partial shapes, where partsare randomly removed from ShapeNet [1] shapes. We aug-ment the ShapeNet training set used by PointNet with ourtraining shapes composed of hybrid shapes, and evaluatethe predicted segmentations on the test set of partial shapes.Fig. 18 shows the results of data augmentation with in-creasing numbers of shapes from our training set, where weevaluate the segmentations by measuring the label accuracy.We observe how each additional batch of shapes helps toimprove the accuracy on the chair set, with a gain of around . The accuracy is also slightly improved for tables, whichtypically have a simpler structure. Moreover, the data pointswith x = 0 represent the accuracy obtained when only theShapeNet training set is used for learning. We notice howthe accuracy is much lower than when our more diverseshapes are used for training, demonstrating that the diverse,plausible shapes do provide additional information for thelearning of the deep network.We confirm this reasoning by visually inspecting the results,such as the set of examples shown in Fig. 19. Note howthe segmentations of these partial shapes have considerableerrors when only the ShapeNet set is used for training. Onthe other hand, we obtain a much refined segmentation,closer to the ground truth, when our evolved shapes areused for data augmentation. For example, the chair-stoolcombo in the second row is missing the side handles, butthe ShapeNet data alone still leads to a segmentation withthis label. In the third row, the leg label appears below theseat, while with our data a better prediction of chair seatsand back rests is obtained. ISCUSSION , L
IMITATIONS , AND F UTURE W ORK
We present the first functionality-aware , user-in-the-loop mod-eling tool to evolve a set of 3D objects, aimed at producinglarge and diverse sets of functionally plausible offsprings.Our work incorporates functionality analysis into the “fitand diverse” set evolution framework of Xu et al. [31].However, rather than restricting part exchange to objectswithin the same category as in Xu et al. [31] and all theworks so far on structure-preserving modeling [20], ourmethod excels at generating cross-category hybrids whileallowing structure breaking. In the end, our evolutionarymodeling tool makes a promising step towards 3D contentgeneration to achieve volume , (intra-class) variety , and (inter-class) variation , producing generations and generations ofwithin- and cross-category hybrids via controlled stochasticshape crossover. But we re-iterate that our modeling toolonly aims to suggest rough design prototypes, not refinedfinal products. Fine-tuned modeling can be applied to se-lected, fully-functional offsprings in post-processing. Hybridizing functionality models . Since cross-category hybridsmay belong to new functional categories beyond those available in our knowledge base, it would be ideal to hy-bridize functionality models. In our current work, we onlyhybridize at the object level, not at the functional categorylevel. Instead of validating the functionality of a new hybridagainst a new functional category it rightfully belongs to,we only compare it partially to known functionality modelssince the hybrid is a crossover between objects covered bythese models. To combine existing functionality models [9]into new models would require sufficient object data andtheir interaction contexts [11]. Furthermore, the resultinghybridized functionality models are generally not unique.
Technical limitations . There is still plenty of room to improveour current method from a technical perspective. Our maintechnical limitations stem from our set goal of only pro-ducing rough design prototypes, where the emphasis hasbeen on functional properties of the offsprings, not theirprecise geometries. Specifically, our evolution operator onlyproduces non-uniform part (group) scaling, while free-formdeformations should allow a richer variety of offsprings.Moreover, our current part connection mechanisms stilllack an understanding of shape semantics and are unableto resolve topological mismatches or merge the meshes ofparts together. At the same time, our current functionalplausibility score and constrained shape evolution only offera starting point for further investigation and development.Last but not the least, we provide the option of comput-ing plausibility scores with a detailed functionality partialmatching. However, the computation relies on an expensivesearch procedure which is slow. Thus, directions for futurework include more efficient manners of detecting partialfunctionalities of shapes, including the use of learning.
Future work . Aside from addressing the above limita-tions, we would also like to explore other functionalitymodeling paradigms. For example, instead of performingfunctionality-preserving style transfer, as in the work byLun et al. [17], we can invert the problem to style-preserving functionality transfer : to transfer the extracted functionalitiesfrom a given shape to another shape, while preservingits stylistic features. Our evolution-based modeling offersa partial solution to this problem. Specifically, a crossoverbetween two shapes is implicitly transferring functionalitiesassociated with the exchanged part groups between theshapes. Another interesting problem to study is functionalanalogy , i.e., to synthesize a new shape S B (cid:48) from an input S B by adding functionalities which would separate a givenshape S A (cid:48) from a given shape S A . R EFERENCES [1] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang,Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, andF. Yu. ShapeNet: An information-rich 3D model repository.
CoRR ,abs/1512.03012, 2015.[2] S. Chaudhuri, E. Kalogerakis, L. Guibas, and V. Koltun. Proba-bilistic reasoning for assembly-based 3D modeling.
ACM Trans. onGraphics , 30(4):35:1–35:10, 2011.[3] S. Chaudhuri and V. Koltun. Data-driven suggestions for creativitysupport in 3D modeling.
ACM Trans. on Graphics , 29(6):183:1–183:9, 2010. [4] D.-Y. Chen, X.-P. Tian, Y.-T. Shen, and M. Ouhyoung. On visualsimilarity based 3D model retrieval. Computer Graphics Forum ,22(3):223–232, 2003.[5] H. Fu, D. Cohen-Or, G. Dror, and A. Sheffer. Upright orientation ofman-made objects.
ACM Trans. on Graphics , 27(3):42:1–42:7, 2008.[6] Q. Fu, X. Chen, X. Su, and H. Fu. Pose-inspired shape synthesisand functional hybrid.
IEEE Trans. on Visualization and ComputerGraphics , 23(12):2574–2585, 2017.[7] T. Funkhouser, M. Kazhdan, P. Shilane, P. Min, W. Kiefer, A. Tal,S. Rusinkiewicz, and D. Dobkin. Modeling by example.
ACMTrans. on Graphics , 23(3):652–663, 2004.[8] R. Hu, M. Savva, and O. van Kaick. Functionality representationsand applications for shape analysis.
Computer Graphics Forum ,37(2):603–624, 2018.[9] R. Hu, O. van Kaick, B. Wu, H. Huang, A. Shamir, and H. Zhang.Learning how objects function via co-analysis of interactions.
ACM Trans. on Graphics , 35(4):47:1–47:12, 2016.[10] R. Hu, Z. Yan, J. Zhang, O. van Kaick, A. Shamir, H. Zhang, andH. Huang. Predictive and generative neural networks for objectfunctionality.
ACM Trans. on Graphics , 37(4):151:1–151:13, 2018.[11] R. Hu, C. Zhu, O. van Kaick, L. Liu, A. Shamir, and H. Zhang.Interaction context (ICON): Towards a geometric functionalitydescriptor.
ACM Trans. on Graphics , 34(4):83:1–83:12, 2015.[12] E. Kalogerakis, S. Chaudhuri, D. Koller, and V. Koltun. A proba-bilistic model for component-based shape synthesis.
ACM Trans.on Graphics , 31(4):55:1–55:11, 2012.[13] V. G. Kim, S. Chaudhuri, L. Guibas, and T. Funkhouser.Shape2Pose: Human-centric shape analysis.
ACM Trans. on Graph-ics , 33(4):120:1–120:12, 2014.[14] V. G. Kim, W. Li, N. Mitra, S. DiVerdi, and T. Funkhouser. Explor-ing collections of 3D models using fuzzy correspondences.
ACMTrans. on Graphics , 31(4):54:1–54:11, 2012.[15] V. Kreavoy, D. Julius, and A. Sheffer. Model composition frominterchangeable components. In
Proc. Pacific Conf. on ComputerGraphics and Applications , pp. 129–138, 2007.[16] H. Lipson and J. B. Pollack. Automatic design and manufacture ofrobotic lifeforms.
Nature , 406(6799):974–978, 2000.[17] Z. Lun, E. Kalogerakis, R. Wang, and A. Sheffer. Functionality pre-serving shape style transfer.
ACM Trans. on Graphics , 35(6):209:1–209:14, 2016.[18] J. Marks, B. Andalman, P. A. Beardsley, W. Freeman, S. Gibson,J. Hodgins, T. Kang, B. Mirtich, H. Pfister, W. Ruml, K. Ryall,J. Seims, and S. Shieber. Design galleries: A general approach to setting parameters for computer graphics and animation. In
Proc.
SIGGRAPH , pp. 389–400, 1997.[19] N. J. Mitra, M. Pauly, M. Wand, and D. Ceylan. Symmetry in 3Dgeometry: Extraction and applications. In
Eurographics State of theArt Reports , pp. 29–51, 2012.[20] N. J. Mitra, M. Wand, H. Zhang, D. Cohen-Or, and M. Bokeloh.Structure-aware shape processing. In
Eurographics State of the ArtReports , pp. 175–197, 2013.[21] D. A. Norman.
The Design of Everyday Things . Basic Books, 2002.[22] M. L. Pilat and C. Jacob. Creature academy: A system for virtualcreature evolution. In
Proc. IEEE Congress on Evolutionary Compu-tation , pp. 3289–3297, 2008.[23] S. Pirk, V. Krs, K. Hu, S. D. Rajasekaran, H. Kang, Y. Yoshiyasu,B. Benes, and L. J. Guibas. Understanding and exploiting objectinteraction landscapes.
ACM Trans. on Graphics , 36(3):31:1–31:14,2017.[24] C. R. Qi, H. Su, K. Mo, and L. J. Guibas. PointNet: Deep learningon point sets for 3D classification and segmentation. In
Proc. IEEEConf. on Computer Vision and Pattern Recognition , pp. 77–85, 2017. [25] M. Savva, A. X. Chang, P. Hanrahan, M. Fisher, and M. Nießner.SceneGrok: Inferring action maps in 3D environments.
ACM Trans.on Graphics , 33(6):212:1–212:10, 2014.[26] H. Shin and T. Igarashi. Magic canvas: Interactive design of a 3-Dscene prototype from freehand sketches. In
Proc. Graphics Interface ,pp. 63–70, 2007.[27] K. Sims. Artificial evolution for computer graphics. In
Proc.SIGGRAPH , pp. 319–328, 1991.[28] K. Sims. Evolving virtual creatures. In
Proc. SIGGRAPH , pp. 15–22,1994.[29] O. Sorkine. Laplacian mesh processing. In
Eurographics State of theArt Reports , pp. 53–70, 2005.[30] K. Xu, V. G. Kim, Q. Huang, and E. Kalogerakis. Data-driven shapeanalysis and processing.
Computer Graphics Forum , 36(1):101–132,2017.[31] K. Xu, H. Zhang, D. Cohen-Or, and B. Chen. Fit and diverse: Setevolution for inspiring 3D shape galleries.
ACM Trans. on Graphics ,31(4):57:1–57:10, 2012.[32] Y. Zheng, D. Cohen-Or, and N. J. Mitra. Smart variations: Func-tional substructures for part compatibility.