SketchGraphs: A Large-Scale Dataset for Modeling Relational Geometry in Computer-Aided Design
SSketchGraphs: A Large-Scale Dataset for ModelingRelational Geometry in Computer-Aided Design
Ari Seff
Princeton University [email protected]
Yaniv Ovadia
Princeton University [email protected]
Wenda Zhou
Columbia University [email protected]
Ryan P. Adams
Princeton University [email protected]
Abstract
Parametric computer-aided design (CAD) is the dominant paradigm in mechanicalengineering for physical design. Distinguished by relational geometry, parametricCAD models begin as two-dimensional sketches consisting of geometric primitives(e.g., line segments, arcs) and explicit constraints between them (e.g., coincidence,perpendicularity) that form the basis for three-dimensional construction operations.Training machine learning models to reason about and synthesize parametricCAD designs has the potential to reduce design time and enable new designworkflows. Additionally, parametric CAD designs can be viewed as instancesof constraint programming and they offer a well-scoped test bed for exploringideas in program synthesis and induction. To facilitate this research, we introduceSketchGraphs, a collection of 15 million sketches extracted from real-world CADmodels coupled with an open-source data processing pipeline. Each sketch isrepresented as a geometric constraint graph where edges denote designer-imposedgeometric relationships between primitives, the nodes of the graph. We demonstrateand establish benchmarks for two use cases of the dataset: generative modelingof sketches and conditional generation of likely constraints given unconstrainedgeometry.
The modern design paradigm for physical objects typically resembles modular programming, wheresimple subcomponents are connected to yield a composed part/assembly with more complex prop-erties [3]. In parametric computer-aided design (CAD), parts generally begin as a collection of 2Dsketches composed of geometric primitives (line segments, circles, etc.) with associated parameters(coordinates, radius, etc.). Primitives and parameters interact via imposed constraints (e.g., equality,symmetry, perpendicularity, coincidence) determining their final configuration. Edits made to anyparameter will propagate along these specified dependencies, updating other properties of the sketchaccordingly. A collection of 3D operations, e.g., extruding a circle into a cylinder, then enable thecreation of solids and surfaces from these 2D sketches.Training machine learning models to construct and reason about object designs has the potentialto enable new and more efficient design workflows. This is an important but challenging domainthat sits at the interface of graphics, relational reasoning, and program synthesis. Recent progress inprobabilistic generative modeling of both continuous (e.g., images and audio [21, 4, 40]) and discrete(e.g., graphs and source code [43, 24, 33]) domains has demonstrated the potential for both samplinghigh-dimensional objects and, in the case of explicit models, estimating their densities/probabilities. a r X i v : . [ c s . L G ] J u l igure 1: Example sketches from the dataset containing at least six geometric primitives. Dashed lines indicate construction geometry, which is used as a reference for other primitives but not physically realized.
If adapted to CAD, such models may be incorporated into the design workflow by, for example,suggesting next steps based on partially specified geometry or offering corrections of implausibleoperations. In addition, if provided with visual observations of a part or sketch, a model may betrained to infer the underlying feature history, allowing for direct modification in CAD software.Beyond the applicability to design itself, reasoning about human-designed structures is fundamentalto artificial intelligence research. One of the challenges of computer vision, for example, is developingrich priors of objects in scenes. It is important to construct priors for the kind of “stuff” encounteredin the world, which may exhibit significant symmetry and modularity, properties that are difficultto capture directly. By developing models for the design process and learning to reason about thecreation of objects, it may become possible to identify hierarchical structures, long-range symmetries,and functional constraints that would be difficult to infer from vision alone. Outside of computervision, a long-horizon goal of artificial intelligence is the task of program induction [25, 18, 7]. Inprogram induction, the objective is to discover computer programs from examples of their inputs andoutputs. Closely related to this are the ideas of programming by demonstration (e.g., Cypher andHalbert [6], Menon et al. [31]) and the problem of program synthesis (e.g., Ellis et al. [9], Gulwaniet al. [15]). Part of the challenge of program induction is identifying a rewarding “sweet spot” in thecoupled space of program representation and induction task; it is difficult to move beyond simpletoy problems. We can view the design of physical objects in a parametric CAD system, however,as an example of constraint programming in which rich geometric structures are specified by animplicit program rather than, e.g., imperatively. It is a highly appealing domain for the study ofprogram induction/synthesis because it is relatively well-scoped and low-order, but clearly requiresthe discovery of modularity to be effective. Moreover, progress on CAD program synthesis andinduction leads to useful tools on a relatively short horizon.We introduce SketchGraphs, a dataset of 15 million sketches extracted from parametric CAD modelshosted on Onshape, a cloud-based CAD platform [1]. Each sketch is represented with the groundtruth geometric constraint graph specifying its construction, where edges denote precise relationshipsimposed by the designer that the must be preserved between specific primitives, the nodes of thegraph.Along with the dataset, we will be releasing an open-source tool suite for data processing, removingobstacles to model development for other researchers.Existing CAD datasets of voxel or mesh-based representations of 3D geometry [5, 42] have enabledwork on sampling realistic 3D shapes [41, 29, 34]. Samples from such models, while impressive, arenot modifiable in a parametric design setting and therefore are not directly usable in most engineeringworkflows. The recent ABC dataset [22] extracts parametric CAD models from Onshape’s publicplatform, as do we, but is geared towards 3D modeling of curves and surfaces, supporting tasks such https://github.com/PrincetonLIPS/SketchGraphs ircleLine Arc M i dp o i n t Line … … P e r p e n d i c u l a r T a n g e n t LineVertical Arc Mirror A n g l e = ° Line Vertical dirX: float dirY: float pntX: float pntY: float start: float end: float c onstruct : bool C o i n c i d e n t C o i n c i d e n t C o i n c i d e n t M i dp o i n t C o i n c i d e n t C o i n c i d e n t C o i n c i d e n t Figure 2:
Example sketch (left) and a portion of its geometric constraint graph (right). Constraints are denotedas edges that either act on a primitive as a whole or some subcomponent of the primitive. Dots represent either aprimitive’s endpoints (left and right dots) or its center point (bottom dot). as patch segmentation and normal estimation. Explicit modeling of the relational structure exhibitedby parametric CAD sketches, the target of SketchGraphs, is currently underexplored.The SketchGraphs dataset may be used to train models directly for various target applications aidingthe design workflow, including conditional completion (autocompleting partially specified geometry)and automatically applying natural constraints reflecting likely design intent (autoconstrain). Inaddition, by providing a set of rendering functions for sketches, we aim to enable work on CADinference from images. Off-the-shelf noisy rendering options allow for sketches to appear hand-drawn.This setup is similar to that proposed in Ellis et al. [10], where TikZ figures are rendered with ahand-drawn appearance to provide training data for a model inferring LaTeX code from images. Here,our sketches are not synthetically produced but rather extracted from real-world parametric CADmodels. See Section 4 for further details on the target applications supported by SketchGraphs.This paper makes the following contributions: • We collect a dataset of 15 million parametric CAD sketches including ground truth geometricconstraint graphs denoting the primitives present and their imposed relationships. • We develop an open-source pipeline for data processing, conversion, and integration withdeep learning frameworks in Python. These technical contributions include: – Specific domain types to enable manipulation of the sketches and their representationsin a structured manner. – A local renderer to enable visualization of sketches obtained from the dataset orgenerated by models. • We establish baseline models for two initial use cases of the dataset: generative modeling ofsketches and inferring likely constraints conditioned on unconstrained geometry.
CAD datasets
Existing large-scale CAD datasets have tended to focus on 3D shape modeling.ModelNet [42] and ShapeNet [5], for example, offer voxel and mesh-based representations of 3Dgeometry, enabling the generative modeling work of Wu et al. [41], Liu et al. [29], Nash et al.[34]. Note that these representations do not store the construction route employed by the originaldesigner of each CAD model. The ABC dataset [22], in contrast to the above, contains parametricrepresentations of 3D CAD models. Like SketchGraphs, the ABC models are obtained from publicOnshape documents. However, the processing pipeline and benchmarks of the ABC dataset support3D modeling of curves and surfaces. In contrast, we extract the 2D sketches that form the basis of 3DCAD models, enabling modeling of the geometric constraints used to ensure design intent.
Sketch datasets
The word sketch is a term of art in the context of CAD, referring specifi-cally to the 2D basis of 3D CAD models, storing both geometric primitives and imposed con-3traints. Large-scale datasets related instead to the colloquial usage of this term have been pro-posed in recent years, specifically focusing on hand-drawn sketches of general image categories(cat, bus, etc.). The QuickDraw dataset was constructed in Ha and Eck [16] from the
Quick,Draw! online game [19] to train an RNN to produce vector image sketches as sequences ofpen strokes. Pairings of pixel-based natural images and corresponding vector sketches are col-lected in the Sketchy dataset [35], intended to train models for sketch-based image retrieval. Likethese datasets based on vector images, our SketchGraphs dataset is also fundamentally focusedon the construction of sketches, not simply their raster appearance. However, here we focus onthe relational geometry underlying parametric CAD sketches, not drawings of general categories.
Primitive type %Line . Circle . Arc . Point . Spline . Ellipse . Constraint type %Coincident . Projected . Distance . Horizontal . Mirror . Vertical . Parallel . Length . Perpendicular . Tangent . Table 1:
Frequencies ofthe most common primitives(top) and constraints (bot-tom).
Graph-structured generative modeling
Modern message passing net-works (e.g., Gilmer et al. [12], Duvenaud et al. [8] extending the earlierwork of Scarselli et al. [36]) have enabled progress in modeling of do-mains that exhibit relational structure, e.g., molecular chemistry andsocial networks. In the context of generative modeling, these networksare often coupled with node and edge-specific prediction modules thatincrementally build a graph [24, 28]. We take a similar approach fortwo example applications demonstrated in Section 4. Several alternativearchitectures have been studied, such as LSTMs on linearized adjacencymatrices [43] or decoding soft adjacency matrices with elements contain-ing probabilities of edge existence [38]. In general, graph modeling issubject to significant representation ambiguity (up to n ! node orderingsfor a graph containing n nodes). Recent work leveraging normalizingflows [27] proposes a permutation-invariant approach for generating nodefeatures. We identify the CAD sketch domain as one which admits a nat-ural ordering over construction operations for the underlying geometricconstraint graphs, which we describe in Section 3. Geometric program induction
Geometric program induction com-prises a practical subset of problems in program induction where the goalis to learn to infer a set of instructions to reconstruct input geometry. For example, Ellis et al. [10]couples a learned shape detector with program search to infer LaTeX code for synthetic imagesof TikZ figures and Sharma et al. [37] trains a reinforcement learning agent to reconstruct simple3D shapes with constructive solid geometry. We view this as a fundamental area of study in orderto develop machine learning models that can aid in design and engineering. Work on generatingprograms interactively, e.g., allowing a model to assess the current program’s output [11], showsparticular promise. Our processing pipeline includes functionality for sketch rendering and queryinga geometric constraint solver to aid research in this direction.
SketchGraphs is aimed towards questions not just concerning the what but in particular the how ofCAD design; that is, not simply what geometry is present but how was it constructed. To this end, weleverage a data source that provides some insight into the actual operations and commands selectedby the designer at each stage of construction. Whereas generic CAD file formats are widely availableonline, e.g., STEP for 3D models and DXF for 2D drawings, these do not store any informationregarding constraints. In recent years, the cloud-based CAD platform Onshape has amassed a largecollection of publicly available models from which detailed construction histories may be queried.For each CAD sketch, we extract ground truth construction operations regarding both the geometricprimitives present and constraints applied to them.
Using Onshape’s API, we gather metadata for all public documents created within a five-year periodfrom 2015 to 2020, leading to over two million unique document IDs. Each document may containmultiple
PartStudios , each specifying the design of an individual component of a CAD model. Wedownload each PartStudio and extract all sketches present, resulting in over 15 million sketches. Note4igure 3:
Left:
Histogram of sketch sizes as measured by the number of primitives and constraints theycontain.
Middle:
Number of constraints as a function of number of primitives in the sketch (opacity representsrespectively median, 69th, 84th and 93rd percentile).
Right:
Average node degree depicted as a function ofsequence position. Note that primitives placed earlier in the sequence tend to serve as common anchors forsubsequent primitives to constrain against. that the PartStudios also contain non-sketch features, e.g., 3D operations, that we do not store inour final dataset. Here, we focus only on the 2D sketches comprising each part and their underlyingconstraint graph representations.To be included in the dataset, each sketch must contain at least one geometric primitive and oneconstraint. The dataset thus ranges from those sketches with larger constraint graphs, which tend tobe more visually interesting, to some very simple sketches, e.g., sketches comprised of a single circle.See Fig. 3 and Table 1 for an overview of the sketch sizes and other dataset statistics.
Geometric constraint graphs offer a succinct representation of 2D CAD sketches. For each sketch,we extract a graph G = ( V, E ) with a set of nodes, V , and a set of edges, E , denoting primitivesand constraints between them, respectively. In general, these are multi-hypergraphs , where multipleedges are permitted to share the same member nodes and each edge may join one or more nodes.When a constraint operates on a single primitive, e.g., a scale constraint such as setting the radius of acircle), we represent the constraint as a loop, an edge connecting the node of interest with itself. Inaddition, hyperedges indicate constraints that operate on three or more nodes. For example, a mirrorconstraint must specify a third primitive to act as an axis of symmetry.Primitives and constraints are described not just by their type, but also by parameters dictating theirbehavior. For primitives, parameters consist of the coordinates denoting their placement within asketch and an isConstruction Boolean indicating if a primitive is to be physically realized (when false)or serve as a reference for other primitives (when true). Note that the initial values of a primitive’scoordinates do not necessarily satisfy any of the constraints present; rather, the task of adjustingprimitives’ coordinates is left to a geometric constraint solver included in standard CAD software.Constraint parameters indicate the primitive(s) acted upon as well as any other numerical or categoricalvalues necessary to fully specify their behavior. For instance, a distance constraint includes a numberindicating the Euclidean distance that two primitives must satisfy. Further details on primitive andconstraint parameters may be found in the appendix.Often, constraints are applied to a specific point on a primitive. For example, two endpoints fromdifferent line segments may be constrained to be a certain distance apart. In order to unambiguouslyrepresent these constraints, we include these sub-primitives as separate nodes in the constraint graph.Best practice in CAD encourages maintaining fully-constrained sketches, meaning a minimallysufficient set of constraints removes all degrees of freedom (DOF) from the sketch primitives [3].This allows for edit propagation and better expression of design intent. Overall, we observe a Pearsoncorrelation coefficient of 0.598 between the total DOF in each sketch and the total DOF removed byconstraints (Fig. 4). Users of SketchGraphs may query for sketches adhering to different thresholdsof constrainedness depending on their application. We exclude constraints directly involving a sketch’s axes in this calculation. Unfortunately, one limitationof our dataset is that these external constraints (constraints involving default geometry not defined by the user)are not currently retrievable via Onshape’s API. This is consistent, however, with the common assumption thatdesigns be fully-constrained up to rigid body transformation. Remaining degrees of freedom F r e q u e n c y Figure 4:
Left:
Histogram of approximated degrees of freedom (DOF) remaining among sketches in theSketchGraphs dataset.
Right:
DOF removed by constraints in a sketch as a function of total DOF from itsprimitives before constraints are applied (opacity represents median, 69th, 84th, and 93rd percentiles).
For certain problem settings, a sequence representation of the constraint graphs may be desired.In generative modeling, for example, graph-structured objects are often modeled autoregressivelyas sequences of construction steps [24, 43]. In the generic case, a canonical node ordering forgraphs may not be available, leading to ambiguity regarding sequential modeling. Here, however,we have access to certain information about each sketch’s construction history that leads to naturalsequence-based representations.In particular, we may access the order in which primitives were added to a sketch by the user,conveying a ground truth node ordering. Rather than being an arbitrary choice among factoriallymany, we observe two trends that support this route: 1) Nodes with greater degree tend to occurearlier, serving as integral building blocks of the sketch (Fig. 3). 2) Adjacent nodes in the orderinghave a greater probability of being adjacent in the graph than randomly selected nodes (0.70 vs. 0.38,respectively).While the ordering of primitives and constraints are both known separately, the relative ordering(interleaving) of primitives and constraints is not recorded in the Onshape models. We canonicalizethe entire sequence by placing an edge’s insertion step immediately following the insertion of itsmember nodes. This emulates the standard design route of constraining primitives as they are addedto a sketch. When there are ties, such as when multiple nodes share more than one edge, or multipleedges share the most recently added node, we may simply revert to the standalone edge ordering.An alternative sequence option stems from the setting where unconstrained geometry is importedinto CAD software, for instance from a drawing scan, and the software attempts to apply intendedconstraints. In this case, the corresponding sequence places all constraints at the end, after specifyingall primitives. We note that there will likely be additional sequence extraction methods of interest tousers of the dataset. Our pipeline may be easily extended to handle custom conversions.
We identify several target applications for which SketchGraphs data may be used to train and developmodels and describe some initial methods to tackle these applications. In addition to mechanicaldesign-focused applications, a domain underexplored in the machine learning community, we notethat these problems share properties with similar tasks in program synthesis and induction. We intendfor SketchGraphs to serve as a test bed for these related lines of work and for the models below toprovide baselines for future research.
CAD packages such as Onshape, AutoCAD, and Solidworks typically contain built-in constraintinferencing. However, these functions are based on manually-defined heuristics, catered towardsinteractive sketching (for example placing a coincidence constraint when a user drags a new primitivefrom an existing one). A sought-after feature is the ability to upload unconstrained geometry, such asfrom a drawing or document scan, and infer the design intent and corresponding set of constraints. Bytreating the primitives in the dataset’s sketches as input, the ground truth constraints may serve as apredictive target. This may be viewed as an instance of program induction in constraint programming.6 riginal Modification A Modification B
True Inferred InferredTrue
Figure 5:
Autoconstraining a sketch. On the left is the original input sketch (only primitives are provided to themodel). Two user modifications are shown with the blue arrows: dragging the top circle’s upwards (modificationA) and both enlarging and dragging it to the right (modification B). Our model correctly picks up on a host ofcoincident, distance, equality, and other constraints. It fails to infer a 7 mm distance constraint between the topand bottom of the rectangle, but the circle correctly maintains midpoint alignment with it (modification A).
The autoconstrain task, then, is to predict a set of constraints given an input configuration of geometricprimitives. Here, we are particularly interested in predicting the “natural” set of constraint that wouldhave been input by a human user. However, we note that other target constraint sets may be considered,by e.g., requiring them to be minimal in some mathematical sense [45, 44, 23]. The autoconstrainproblem can also be viewed as an example of a link prediction task in which the induced relationshipsare the constraints; see e.g., Taskar et al. [39], Liben-Nowell and Kleinberg [26], Lü and Zhou [30].
We propose an auto-regressive model based on message passing networks (MPNs) [8, 12] whereinformation about the input geometric primitives is propagated along a growing set of constraints,iteratively building a constraint graph. The model is tasked with predicting the sequence of edgescorresponding to the given node sequence, and proceeds in a recurrent fashion by iteratively predictingthe next edge (represented as a pair of the edge partner node and the edge label) for each node (or a stop token indicating to move to the next node).At inference time, the model is additionally given a mask indicating (approximately) which constraintsare satisfied in the sketch (these may be satisfied because they were originally imposed in the dataset,or they may be a consequence of the original constraints). This ensures that the model only selectsvalid constraints and does not deform the sketch under consideration.
We train this model on a subset of SketchGraphs of 2.2 million sketches limited to the most commontypes of primitives (Point, Line, Circle, and Arc) and at most 16 primitive primitives per sketch.We exclude hypergraphs from consideration here and only model two-node and single-node edges.50K held-out sketches are used for testing. A full description of the model architecture, the trainingprocedure and the inference procedure is available in the supplementary material.The autoconstrain model is evaluated by predicting edges on a test dataset. We obtain an averageedge precision of . and an average edge recall of . . The average F1 score is . . We alsoevaluate the test negative log-likelihood at an average of . bits per edge. For reference, a uniformchoice among valid constraints scores an average entropy of . bits per edge.We also demonstrate the inferred constraints qualitatively by editing a test sketch and observing theresulting solved state Fig. 5. A variety of target tasks may be approached under the broader umbrella of generative modeling.By learning to predict sequences of sketch construction operations, for example, models may beemployed for conditional completion, interactively suggesting next steps to a CAD user. In addition,7 xplicit generative models, estimating probabilities (or densities) of examples, may be used to assessthe overall plausibility of a sketch via its graph or construction sequence, offering corrections ofdubious operations (similar to “autocorrect”). This also provides a path to a CAD analog of inductiveprogramming support that has been deployed in Microsoft Excel [14].Here we develop an initial model and benchmark for unconditional generative modeling. We train amodel on full construction sequences for the sketch graphs, both nodes and edges. While we modelconstraint parameters (edge features), we only model the primitive type parameter for the nodes,leaving the task of determining the final configuration of primitive coordinates to a constraint solver.
The model resembles that from the autoconstrain task above, with an additional node-adding module.When edge sampling for a given node has completed, the model outputs a distribution over possibleprimitive types to add to the graph. After a node’s insertion, any sub-primitive nodes associatedwith the new node (e.g., endpoints of a line segment) are deterministically added to the graph withcorresponding edges denoting the sub-primitive relationship. Alternatively, a stop token may beselected that ceases graph construction. See supplementary for further model details.
The evaluation of high-dimensional generative models is an open problem. Here we provide quantita-tive evaluation consisting of likelihood on the held-out test set and include distributional statisticsin the supplementary. Using the same train-test split as in Section 4.1.2, the average negative log-likelihood of test examples according to the trained model is . bits per graph. In comparison, astandard LZMA compressor, applied to a short canonical representation of the data, yields an averageentropy of . bits per sketch. Full details of the evaluation are available in the supplementarymaterial along with some renderings of generated sketches. A highly-sought feature for CAD software is the ability to input a noisy observation of an object(2D drawing, 3D scan, etc.) and infer its design steps, producing a plausible parametric CAD model.Inferring sketch primitives and constraints, which form the 2D basis of 3D CAD models, is an integralcomponent of this application. Our pipeline includes rendering functions for producing images of thesketches, including noisy rendering to simulate a hand-drawn appearance. This allows generatingmillions of training pairs of rendered sketches and corresponding geometric constraint graphs.
New models trained on the SketchGraphs data can lead to vectorial latent representations that captureimportant semantic content in sketches. Such vector representations have been developed for naturallanguage processing [2, 32], speech recognition [17], computer vision [20], and computational chem-istry [13]. These representations have opened up a space of creative new possibilities for downstreamtasks from search to content recommendation. Unsupervised learning on the SketchGraphs dataenables such possibilities for CAD designs.
This paper has introduced SketchGraphs, a large-scale dataset of parametric CAD sketches andprocessing pipeline intended to facilitate research in ML-aided design and broader problems inrelational reasoning and program induction. Each sketch is accompanied by the ground truthgeometric constraint graph denoting its configuration. We demonstrate two use cases of the dataset,unconditional generative modeling and conditionally inferring constraints given primitives, providinginitial benchmarks for these applications.While we focus on 2D sketches here, which serve as the anchors for full parametric CAD models,future work will aim to make the complete set of construction operations accessible to modeling. We8ill also be providing benchmarks for additional applications supported by the dataset, includingparametric CAD inference from images, a potentially powerful design aid.
Acknowledgments and Disclosure of Funding
We would like to thank Onshape for the CAD sketches and API support. We also thank AlexBeatson, Daniel Suo, Gregory Gundersen, and members of the Princeton Laboratory for IntelligentProbabilistic Systems for valuable discussion and feedback. This work was partially funded by NSFIIS-1421780 and the DataX Program at Princeton University through support from the SchmidtFutures Foundation. AS was supported by the National Defense Science and Engineering GraduateFellowship (NDSEG) Program.
References [1] Onshape. . Accessed: 2019-08-01.[2] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilisticlanguage model.
Journal of Machine Learning Research , 3(Feb):1137–1155, 2003.[3] Bernhard Bettig and Christoph M Hoffmann. Geometric constraint solving in parametricComputer-Aided Design.
J. Comput. Inf. Sci. Eng. , 11(2):021001, June 2011.[4] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelitynatural image synthesis. In
International Conference on Learning Representations , 2019.[5] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, ZimoLi, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu.ShapeNet: An Information-Rich 3D Model Repository. Technical Report arXiv:1512.03012[cs.GR], 2015.[6] Allen Cypher and Daniel Conrad Halbert.
Watch What I Do: Programming by Demonstration .MIT press, 1993.[7] Eyal Dechter, Jon Malmaud, Ryan P Adams, and Joshua B Tenenbaum. Bootstrap learningvia modular concept discovery. In
Twenty-Third International Joint Conference on ArtificialIntelligence , 2013.[8] David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Gómez-Bombarelli, TimothyHirzel, Alan Aspuru-Guzik, and Ryan P Adams. Convolutional networks on graphs for learningmolecular fingerprints. In
Advances in Neural Information Processing Systems , 2015.[9] Kevin Ellis, Armando Solar-Lezama, and Josh Tenenbaum. Unsupervised learning by programsynthesis. In
Advances in Neural Information Processing Systems , pages 973–981, 2015.[10] Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Josh Tenenbaum. Learning to infergraphics programs from hand-drawn images. In
Advances in Neural Information ProcessingSystems 31 . 2018.[11] Kevin Ellis, Maxwell Nye, Yewen Pu, Felix Sosa, Josh Tenenbaum, and Armando Solar-Lezama.Write, execute, assess: Program synthesis with a repl. In
Advances in Neural InformationProcessing Systems 32 . 2019.[12] Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl.Neural message passing for quantum chemistry. In
International Conference on MachineLearning , 2017.[13] Rafael Gómez-Bombarelli, Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato,Benjamín Sánchez-Lengeling, Dennis Sheberla, Jorge Aguilera-Iparraguirre, Timothy D Hirzel,Ryan P Adams, and Alán Aspuru-Guzik. Automatic chemical design using a data-drivencontinuous representation of molecules.
ACS central science , 4(2):268–276, 2018.914] Sumit Gulwani, José Hernández-Orallo, Emanuel Kitzelmann, Stephen H Muggleton, UteSchmid, and Benjamin Zorn. Inductive programming meets the real world.
Communications ofthe ACM , 58(11):90–99, 2015.[15] Sumit Gulwani, Oleksandr Polozov, Rishabh Singh, et al. Program synthesis.
Foundations andTrends in Programming Languages , 4(1-2):1–119, 2017.[16] David Ha and Douglas Eck. A neural representation of sketch drawings. In
InternationalConference on Learning Representations , 2018.[17] Mikael Henaff, Kevin Jarrett, Koray Kavukcuoglu, and Yann LeCun. Unsupervised learning ofsparse features for scalable audio classification. In
ISMIR , volume 11, page 2011, 2011.[18] Irvin Hwang, Andreas Stuhlmüller, and Noah D Goodman. Inducing probabilistic programs byBayesian program merging. arXiv preprint arXiv:1110.5667 , 2011.[19] Jonas Jongejan, Henry Rowley, Takashi Kawashima, Jongmin Kim, and Nick Fox-Gieg. TheQuick, Draw! - A.I. experiment. https://quickdraw.withgoogle.com , 2016.[20] Koray Kavukcuoglu, Marc’Aurelio Ranzato, Rob Fergus, and Yann LeCun. Learning invariantfeatures through topographic filter maps. In , pages 1605–1612. IEEE, 2009.[21] Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions.In
Advances in Neural Information Processing Systems 31 . 2018.[22] Sebastian Koch, Albert Matveev, Zhongshi Jiang, Francis Williams, Alexey Artemov, EvgenyBurnaev, Marc Alexa, Denis Zorin, and Daniele Panozzo. ABC: A big CAD model datasetfor geometric deep learning. In
Proceedings of the IEEE Conference on Computer Vision andPattern Recognition , 2019.[23] Ming Li, Frank C. Langbein, and Ralph R. Martin. Detecting design intent in approximateCAD models using symmetry.
Computer-Aided Design , 42(3):183 – 201, 2010.[24] Yujia Li, Oriol Vinyals, Chris Dyer, Razvan Pascanu, and Peter Battaglia. Learning deepgenerative models of graphs. In
International Conference on Machine Learning , 2018.[25] Percy Liang, Michael I Jordan, and Dan Klein. Learning programs: A hierarchical Bayesianapproach. In
Proceedings of the 27th International Conference on Machine Learning , pages639–646, 2010.[26] David Liben-Nowell and Jon Kleinberg. The link-prediction problem for social networks.
Journal of the American society for information science and technology , 58(7):1019–1031,2007.[27] Jenny Liu, Aviral Kumar, Jimmy Ba, Jamie Kiros, and Kevin Swersky. Graph normalizingflows. In
Advances in Neural Information Processing Systems 32 . 2019.[28] Qi Liu, Miltiadis Allamanis, Marc Brockschmidt, and Alexander Gaunt. Constrained graphvariational autoencoders for molecule design. In
Advances in Neural Information ProcessingSystems . 2018.[29] Shikun Liu, Lee Giles, and Alexander Ororbia. Learning a hierarchical latent-variable model of3d shapes. In . IEEE, 2018.[30] Linyuan Lü and Tao Zhou. Link prediction in complex networks: A survey.
Physica A:statistical mechanics and its applications , 390(6):1150–1170, 2011.[31] Aditya Menon, Omer Tamuz, Sumit Gulwani, Butler Lampson, and Adam Kalai. A machinelearning framework for programming by example. In
International Conference on MachineLearning , pages 187–195, 2013.[32] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre-sentations of words and phrases and their compositionality. In
Advances in neural informationprocessing systems , pages 3111–3119, 2013.1033] Vijayaraghavan Murali, Letao Qi, Swarat Chaudhuri, and Chris Jermaine. Neural sketch learningfor conditional program generation. In
International Conference on Learning Representations ,2018.[34] Charlie Nash, Yaroslav Ganin, S. M. Ali Eslami, and Peter W. Battaglia. Polygen: An autore-gressive generative model of 3d meshes, 2020.[35] Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays. The sketchy database:Learning to retrieve badly drawn bunnies.
ACM Transactions on Graphics (proceedings ofSIGGRAPH) , 2016.[36] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini.The graph neural network model.
IEEE Transactions on Neural Networks , 20:61–80, 2009.[37] Gopal Sharma, Rishabh Goyal, Difan Liu, Evangelos Kalogerakis, and Subhransu Maji. Csgnet:Neural shape parser for constructive solid geometry. In
The IEEE Conference on ComputerVision and Pattern Recognition (CVPR) , 2018.[38] Martin Simonovsky and Nikos Komodakis. GraphVAE: Towards generation of small graphsusing variational autoencoders. In
ICAN , 2018.[39] Ben Taskar, Ming-Fai Wong, Pieter Abbeel, and Daphne Koller. Link prediction in relationaldata. In
Advances in Neural Information Processing Systems , pages 659–666, 2004.[40] Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, AlexanderGraves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generativemodel for raw audio. In
Arxiv , 2016. URL https://arxiv.org/abs/1609.03499 .[41] Jiajun Wu, Chengkai Zhang, Tianfan Xue, William T Freeman, and Joshua B Tenenbaum.Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling.In
Advances in Neural Information Processing Systems , 2016.[42] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao. 3d shapenets: A deeprepresentation for volumetric shapes. In
Computer Vision and Pattern Recognition , 2015.[43] Jiaxuan You, Rex Ying, Xiang Ren, William Hamilton, and Jure Leskovec. GraphRNN:Generating realistic graphs with deep auto-regressive models. In
International Conference onMachine Learning , 2018.[44] Gui-Fang Zhang. Well-constrained completion for under-constrained geometric constraintproblem based on connectivity analysis of graph. In
Proceedings of the 2011 ACM Symposiumon Applied Computing , SAC ’11, page 1094–1099, New York, NY, USA, 2011. Association forComputing Machinery. ISBN 9781450301138. doi: 10.1145/1982185.1982427.[45] Gui-Fang Zhang and Xiao-Shan Gao. Well-constrained completion and decomposition forunder-constrained geometric constaint problems.
International Journal of ComputationalGeometry & Applications , 16(05n06):461–478, 2006. doi: 10.1142/S0218195906002142.11
Primitive Parameters
Primitives are accompanied by both numerical and Boolean parameters specifying their initialpositions within a sketch. For certain primitives, Onshape and other CAD programs employ anoverparameterized description that aids in constraint solving. In our pipeline, the classes representingeach primitive type contain attributes corresponding to Onshape’s parameterization but includemethods for conversion to standard parameterizations.As mentioned in the main text, all primitives have an isConstruction
Boolean parameter indicating ifa primitive is to be physically realized or simply serve as a reference for other primitives. We providethe remaining parameterization for common primitive types below.
Point (dof: 2) • x ( float ): x coordinate • y ( float ): y coordinate Line (dof: 4) • dirX ( float ): x component of unit direction vector • dirY ( float ): y component of unit direction vector • pntX ( float ): x coordinate of any point on the line • pntY ( float ): y coordinate of the same point as above • startParam ( float ): signed distance of starting point relative to ( pntX , pntY ) • endParam ( float ): signed distance of ending point relative to ( pntX , pntY ) Circle (dof: 3) • xCenter ( float ): x coordinate of circle center • yCenter ( float ): y coordinate of circle center • xDir ( float ): x component of unit direction vector • yDir ( float ): y component of unit direction vector • radius ( float ): radius • clockwise ( bool ): orientation of the unit direction vector Arc (dof: 5) • xCenter ( float ): x coordinate of corresponding circle’s center • yCenter ( float ): y coordinate of corresponding circle’s center • xDir ( float ): x component of unit direction vector • yDir ( float ): y component of unit direction vector • radius ( float ): radius of corresponding circle • clockwise ( bool ): orientation of the unit direction vector • startParam ( float ): starting angle relative to unit direction vector • endParam ( float ): ending angle relative to unit direction vector Ellipse (dof: 5) • xCenter ( float ): x coordinate of ellipse’s center • yCenter ( float ): y coordinate of ellipse’s center • xDir ( float ): x component of unit direction vector • yDir ( float ): y component of unit direction vector • radius ( float ): greater (major) radius • minorRadius ( float ): smaller radius • clockwise ( bool ): orientation of the unit direction vector Circles are considered to have an angular direction in order to account for rotation of sketch componentsinvolving circles. Constraint Parameters
All constraints act on at least one primitive, indicated by the corresponding edge’s member nodes.A subset of constraints require additional numerical, enumerated, or Boolean parameters to fullyspecify their behavior. Here we list the general parameters that may accompany constraints followedby the schemata for common constraint types. We exclude any external constraints (e.g., constraintsinvolving projected geometry) and describe only those that act on user-defined geometry within asketch. Numerical parameters follow user-specified units.Note that constraint parameters are considered internal to Onshape and thus external documentationis sparse. We determine parameter functionality based on discussions with Onshape developers andusage of the solver.
B.1 Parameters • local ( reference ): a reference to a primitive. A constraint may have one or more ofthese (e.g., local0 , local1 , ...). The alternative parameter names localFirst and localSecond are used interchangeably in the data with local0 and local1 , respectively. • length ( float ): quantity for a numerical constraint (e.g., a 3 cm distance) • angle ( float ): quantity for an angular constraint (e.g., 45 degrees) • clockwise ( bool ): orientation of an angular constraint • aligned ( bool ): whether the start and end points of primitives in an angular constraint arealigned in the angle computation • direction ( enum ): the measurement type for a distance value (must be on of minimum , vertical , or horizontal ) • halfSpace ( enum ): the relative positioning to be maintained by primitives in distance-based constraints (must be one of left or right ) B.2 Schemata
Below, we list each parameter schema and the constraints adhering to it. Note that some constraintscan appear with more than one schema. For example, a horizontal constraint may act on a singleprimitive (specifying only local0 ) or two primitives (specifying both local0 and local1 ). Numer-ical constraints are listed here with their most frequent schema, although a few other schemas mayappear in the dataset.( local0 )Horizontal, Vertical( local0 , local1 )Coincident, Horizontal, Vertical, Parallel, Perpendicular, Tangent, Midpoint, Equal, Offset, Concentric( local0 , local1 , local2 )Mirror( local0 , length )Diameter, Radius( local0 , direction , length )Length( local0 , local1 , direction , halfSpace0 , halfSpace1 , length )Distance( local0 , local1 , aligned , clockwise , angle )Angle B.3 Numerical parameter distributions
We examine the values observed for the two constraint parameters specifying a quantity: length and angle . As described above, the length parameter is used in several numerical constraints.13ength %5 mm . . . . . . . . . . Angle (deg) %45 . . . . . . . . . . Table 2:
Frequencies of the most common values observed for length (left) and angle (right) parameters. Allvalues are converted to common units for frequency computation. For length , we display the values in the unitsrequiring the fewest digits. Note that although the standalone
Perpendicular constraint is generally used for90-degree angles, perpendicularity is sometimes imposed with an angular constraint as seen here.
See Table 2 for the frequencies of the most common parameter values. Unsurprisingly, the mostcommon angles tend to evenly divide 360 degrees. The most common length parameters tend tocorrespond to standard sizes of common parts (e.g., a 5 mm screw). Fig. 6 displays the cumulativefrequency of parameter values when sorted from most to least frequent. The 300 most frequent valuesfor angle and length account for 95.8% and 82.1% of all occurrences, respectively. angle , as ascale-invariant parameter, exhibits a bit less diversity than length . C u m u l a t i v e f r e q u e n c y anglelength Figure 6:
Cumulative frequency of unique parameter values when sorted from most to least frequent.
C Example sketch constructions
Below, we render the construction steps for some of the example sketches in Fig. 1 according to theuser-defined primitive orderings. 14igure 7:
Construction of a dataset sketch proceeding from top left to bottom right.
Figure 8:
Construction of a dataset sketch proceeding from top left to bottom right.
Construction of a dataset sketch proceeding from top left to bottom right.
Figure 10:
Construction of a dataset sketch proceeding from top left to bottom right. Autoconstrain Model
D.1 Task description
We train the model to predict the graph edges in sequence following the ground truth orderingdescribed in Section 3.3. The model is trained in a supervised fashion simultaneously on two relatedtasks: • (partner prediction) Given the nodes in the sketch, and the graph representing the constraintsat the given step of the construction sequence, predict which node should be attached tothe current node in order to create a new constraint (or choose to move to the next node bypredicting a sentinel node to attach). • (constraint label prediction) Given the above, and the target partner of the current constraint,predict the type of the current constraint. D.2 Model description
The model may conceptually be divided into three components: 1) an input representation component,responsible for embedding the features from the primitives and constraints, 2) a message-passingcomponent, responsible for transforming these features using the graph structure, and 3) a readoutcomponent, which outputs probabilities for the specific tasks according to the transformed features.We describe each one in turn.Our model is parametrized by a global complexity parameter m (we use m = 384 in the results wepresent). Input representation
Constraints are only identified by their referenced primitives, and their type.At the current stage, we only embed their type using a m -dimensional embedding layer. Primitivesare a sequence of heterogenous discrete elements, and thus require a little more care. The type ofthe primitive (i.e. Point, Line etc.) is embedded similarly using a m -dimensional embedding layer.However, each primitive type may have a different number of parameters, which must representedsimilarly. Continuous parameters are quantized and represented by their quantized value, which isembedded using a m -dimensional layer. For a given primitive, all its parameters are embedded andaveraged to form a parameter embedding of size m (if a primitive has no parameters, this parameterembedding is set to zero). The embedding for the primitive is then computed by concatenating theembedding corresponding to the type, and that of the parameters, and projecting this m vector ontoone of size m through a dense layer. Message passing
Prior to message passing, the node embeddings are transformed using 3-layerrecurrent neural network using the GRU architecture (and with the hidden size set to m ). The nodeembeddings are then recursively transformed using a message passing network, such as at stage s , weperform the update: a ( s +1) v = (cid:88) u :( u,v ) ∈ E f e ( m ( s ) u , c ( u,v ) ) , m ( s +1) v = f n ( a ( s +1) v , m ( s ) v ) , where here c ( u, v ) denotes the representation for the constraint computed previously, and m (0) u is therepresentation for the primitive computed previously. We set f e as a linear layer that concatenates m ( s ) u and c ( u,v ) and set f n to take the functional form of a GRU cell. We use 3 message passingsteps in our presented results.A global representation for the graph is also computed by computing a weighted of the final nodemessages, where the weights are computed using a learned function comprised of a sigmoid appliedto a linear layer. This representation is combined with the final state of the GRU using a linear layeragain to obtain a final global representation for the problem. Readout
The readout for predicting the partner takes as input the final node representations for eachnode in the graph, the final node representation for the current node, and the global representation.17hese representations are concatenated and fed to a fully-connected two-layer network (with ReLUnon-linearity) which produces a scalar value for each node in the graph. This value is interpreted asan unnormalized log-probability for the partner selected being the given node, where an implicit 0value is given to a sentinel node representing a request to move to the next node.The readout for predicting the label takes as input the final node representations for the current nodeand the partner node, as well as the global representation. These representations are concatenated andfed into a three layer fully-connected neural network. The output of that network is interpreted asunnormalized log-probabilities for predicted edge type.
D.3 Training
The model is trained on a subset of the dataset consisting of about 2.2M sketches, filtered such thatthe largest sketch does not exceed 16 primitives. The model is presented with random edges (or “stop”edges representing a request to move to the next node), selected uniformly among all possible edgesin the sketches (this implies that the sketches are weighted towards longer sketches). We train themodel using the Adam optimizer with a batch size of 8192 and a learning rate of − , where thelearning rate is understood to apply to the loss as expressed as a sum over the batch (rather than anaverage). The training is performed over 150 epochs of the data, with the learning rate decaying by afactor of 10 at epochs 50 and 150. The training is performed on a server equipped with 4 Nvidia TitanX (Pascal) and dual Intel Xeon E5-2667 v4 (total 32 logical cores) and takes 3 hours and 30 minutes. D.4 Evaluation
We evaluate the model on a separate held-out set of 50K sketches. During the evaluation stage, invalidedge predictions are masked from the model. Although the model naturally operates in a factorizedfashion: P ( edge | X ) = P ( edge type | edge partner , X ) P ( edge partner | X ) we note that the mask itself does not factorize in the given fashion (as the validity of a partner maydepend on the specific constraint being considered). We thus reconstruct the full joint distribution onthe edge partner and labels according to the given conditionals, and mask and re-scale the predictionsusing the joint distribution directly. As reported in the main text, we obtain an average edge recall of . ± . and an average edge precision of . ± . (where the brackets indicate the standarderror of the estimate). E Generative Model
E.1 Model description
The generative model shares many similarities with the autoconstrain model described in Ap-pendix D.2, and thus only major differences are highlighted here.
Input representation
The input representation is similar to that of the autoconstrain model, withthe exceptions that primitive parameters are ignored whereas constraint parameters are represented.
Message passing
The message passing process is identical to the autoconstrain model, except norecurrent model is used, and the node embeddings are used directly.
Readout
In addition to the readout networks presented for the autoconstrain model, additionalreadout models are present for the primitive prediction task and predicting constraint parameters. Theprimitive prediction readout is given the global embedding for the graph, and is tasked to predict thetype of the next primitive to be added to the construction sequence. It is implemented as a 3 layerfully-connected neural network. The constraint feature readout is given the constraint type, as well asa representation computed from the representation of the primitives participating in the constraint.The features are then read-out sequentially using a recurrent neural network.18 .2 Training
The model is trained on the same data as the autoconstrain model. Numerical constraint parametersadhering to the most frequent schemas (Appendix B) are included as targets. The model is trainedusing the Adam optimizer, with a batch size of 6144 and a learning rate of − , where the learningrate is understood to apply to the loss as expressed as a sum over the batch. The training is performedfor 150 epochs of the data, with the learning rate decaying by a factor of 10 at epochs 50 and 150.The training is performed on a server equipped with 4 Nvidia Titan X (Pascal) and dual Intel XeonE5-3667 v4 (total 32 logical cores), although only 3 GPUs were used due to a CPU bottleneck. Thetraining takes 3 hours and 50 minutes. E.3 Evaluation
The evaluation is performed on the same testing split as the autoconstrain model.
LZMA comparison
To estimate the performance of the LZMA compressor on the data, werepresent the construction sequence as a sequence of integers (representing all labels and features).Such sequences are then concatenated and compressed using Python’s implementation of LZMA withpreset 9 and
LZMA_EXTREME . Let s n denote the compressed size (in bits) of the the first n elements ofthe dataset, we report an estimate of the average entropy rate by computing ( s − s ) / . Distributional statistics
We compare a host of statistics for the ground-truth training dataset tostatistics on 10K generated samples. Fig. 12 depicts sketch-size distributions in terms of primitive andconstraint counts, and Fig. 13 depicts the distribution of degrees of freedom in each set of sketches.Error bars in these histograms represent 5th and 95th percentiles acquired by resampling the generatedsketches with replacement 2K times. We also compare the distribution of primitive and constrainttypes in Fig. 14.For depictions of generated sketches, see Fig. 11. While this baseline model produces primitive types,constraint types, and constraint parameters, it is not trained to initialize the primitive coordinates. Thesolver determines the final configuration starting from a uniform initialization, which limits the visualdiversity of the observed samples. Future work will include modeling the primitive coordinates.Figure 11:
Random samples from the trained generative model containing at least two primitives. A solver isused to determine the final configuration of the sketch after the model samples the geometric constraint graph.Each primitive is initialized uniformly (e.g., all lines initially lie on the x-axis from 0 to 1) and their coordinatesare updated by the solver. This baseline model, which does not output primitive parameters, is able to capturesome of the patterns observed for small sketches but does not produce many sophisticated sketches. F r e q u e n c y samplesdataset 0 10 20 30 40 50Number of constraints0.000.050.100.15 F r e q u e n c y samplesdataset Figure 12:
Distributions of sampled and training set sketch sizes. Error bars represent bootstrapped 5th and95th percentiles. F r e q u e n c y samplesdataset 0 10 20 30 40 50 60DOF removed by constraints0.000.050.100.150.20 F r e q u e n c y samplesdataset 0 5 10 15 20 25 30 35 40Net DOF0.000.050.100.150.200.25 F r e q u e n c y samplesdataset Figure 13:
Degree-of-freedom statistics for sampled and training set sketches. Error bars represent bootstrapped5th and 95th percentiles.
Figure 14: