Computational LEGO Technic Design
CComputational LEGO ® Technic Design
HAO XU, KA-HEI HUI, and CHI-WING FU,
The Chinese University of Hong Kong
HAO ZHANG,
Simon Fraser University
Fig. 1. We introduce a fully automatic method to compute LEGO ® Technic designs (c & d) that resemble user input sketches, which may be annotated withjoint rotations (bottom-left). The designs are built by coherently-connected LEGO ® Technic bricks (b) that respect the symmetry and structural integrity of themodels, as well as the dynamics implied at the specified joints, while aiming for minimalistic layouts.
We introduce a method to automatically compute LEGO ® Technic modelsfrom user input sketches, optionally with motion annotations. The generatedmodels resemble the input sketches with coherently-connected bricks andsimple layouts, while respecting the intended symmetry and mechanicalproperties expressed in the inputs. This complex computational assemblyproblem involves an immense search space, and a much richer brick setand connection mechanisms than regular LEGO ® . To address it, we firstcomprehensively model the brick properties and connection mechanisms,then formulate the construction requirements into an objective function,accounting for faithfulness to input sketch, model simplicity, and structuralintegrity. Next, we model the problem as a sketch cover, where we iterativelyrefine a random initial layout to cover the input sketch, while guided bythe objective. At last, we provide a working system to analyze the balance,stress, and assemblability of the generated model. To evaluate our method, wecompared it with four baselines and professional designs by a LEGO ® expert,demonstrating the superiority of our automatic designs. Also, we recruitedseveral users to try our system, employed it to create models of varyingforms and complexities, and physically built most of them.CCS Concepts: • Computing methodologies → Shape modeling .Additional Key Words and Phrases: LEGO ® , Technic series, computationaldesign, fabrication, assembly LEGO ® is a trademark of the LEGO ® Group, which does not sponsor, authorize orendorse this work. All information in this paper is collected and interpreted by itsauthors and does not represent the opinion of the LEGO Group. Permission (images inpaper) granted by LEGO ® . All rights reserved.Authors’ addresses: Hao Xu; Ka-Hei Hui; Chi-Wing Fu, The Chinese University ofHong Kong; Hao Zhang, Simon Fraser University.Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from [email protected].© 2019 Association for Computing Machinery.0730-0301/2019/11-ART196 $15.00https://doi.org/10.1145/3355089.3356504 ACM Reference Format:
Hao Xu, Ka-Hei Hui, Chi-Wing Fu, and Hao Zhang. 2019. ComputationalLEGO ® Technic Design.
ACM Trans. Graph.
38, 6, Article 196 (Novem-ber 2019), 14 pages. https://doi.org/10.1145/3355089.3356504
The LEGO ® Technic system [The LEGO ® Group 2018b] was intro-duced as an expert series for building advanced 3D models in 1977.With brick pieces well beyond those in regular LEGO ® , e.g., beams,axles, pins, and connectors, as shown in Figure 1(b), one can build 3Dframe-based structures like those commonly-seen in architecture, aswell as mechanical assemblies that exhibit dynamic behavior suchas joint rotations. LEGO ® Technic designs have resulted in a varietyof customizable robotics and mechanical models; see Figure 2 forsome elaborative designs by enthusiasts.Designing LEGO ® Technic models is considerably more challeng-ing than regular LEGO ® models, even without adding complex me-chanical elements such as gears and pulleys. Compared with regularLEGO ® bricks, which are mostly connected through studs on topof the bricks, LEGO ® Technic bricks are connected in a variety ofways, also via different kinds of pins and connectors, some of whichallow joint rotations, as shown in Figures 1(b) and 5. The sheernumber of assembly varieties leads to an immense search space.For instance, there are over six billion ways of assembling a simplesquare with nine-unit long sides. Also, as a result of the connectionmechanisms, LEGO ® Technic models are often built with stricter andmore intricate assembly order than regular LEGO ® models. Yet, theTechnic system can better adapt to form non-blocky, frame-based,and even articulable shapes in 3D, since the LEGO ® Technic beamscan be arranged flexibly in different orientations for building dif-ferent parts in the models; see the green arrows in Figure 2(b) thatindicate the orientations of the associated beams.
ACM Trans. Graph., Vol. 38, No. 6, Article 196. Publication date: November 2019. a r X i v : . [ c s . G R ] J u l Fig. 2. LEGO ® Technic models designed by enthusiasts: (a) a guitar robotby YouTuber TECHNICally Possible (see “ ” with over “20.6M” views); (b) a mechanical loom by N.Lespour (see “ ”); and (c)a 3D LEGO ® printer coined MakerLegoBot by W. Gorman (see “ ”).Fig. 3. A cross can be built with different beam layering options. Due to the connection mechanisms using pins and connectors,Technic models have an entirely different and far more complexbuilding style compared with the simple bottom-up style of regularLEGO ® brick assemblies. LEGO ® Technic builders have to mindfullyplan the beam placements , orientations , connections , and layering (seeFigure 3). Typically, they need to think several steps ahead and createsmall assemblies to test the feasibility via trial-and-error [Kmieć2016; The LEGO ® Group 2018b]. Given such complexity, manualdesigns of Technic models are tedious and challenging, requiringsubstantial expertise and time to realize a working design.In this work, we aim to develop a fully automatic computationalmethod for LEGO ® Technic model construction. Specifically, given auser-drawn line sketch of a frame-based 3D model, optionally withannotations of joint rotations, e.g., see Figure 1(a), our method auto-matically selects and arranges LEGO ® Technic bricks to form Technicassemblies that resemble and cover the input sketches, while respectingthe structural integrity and joint motions of the designs and strivingfor simplicity and cost-effectiveness of the assemblies.
The computational challenges of this problem stem from theimmense search space and the multitude of different connectionmechanisms, beam orientations, and layering options. A viable con-struction often involves much more than aligning the Technic pieceswith the sketched lines. To achieve a coherent, structurally plausi-ble, and functional assembly, the final LEGO ® Technic models maydeviate in nontrivial ways from the input sketches, e.g., see thecomputed assembly at the tail of the airplane in Figure 1.To approach the problem, we first comprehensively model Tech-nic constructions by enumerating the brick properties and connec-tion mechanisms, conceptualizing the input as a guiding graph (seeFigures 4 (a,b)), and modeling the construction requirements into anobjective function. Further, we formulate the brick layout problemas a sketch cover and solve it by first estimating the local beam ori-entations. Then, we further iteratively refine a random initial layoutinto a coherent model to cover the input sketch, while guided bythe objective function; see Figures 4 (c,d).Overall, our work makes the following contributions: • the first method that automatically constructs LEGO ® Technicmodels that are coherently-connected, assemblable, and func-tional, based on the user-provided model specifications; • a computational model for Technic constructions, consideringvarious brick properties and types, connection mechanisms,coherency, and construction requirements, including supportfor joint rotations, a natural and fundamental mechanicalfunctionality provided by LEGO ® Technic designs; and • finally, a working system, which enables user input designs,provides structural integrity analysis, and produces assemblyinstructions and visualizations of the assembly process tofacilitate physical constructions (see Figures 4 (e,f,g)).We demonstrate our method by generating Technic models ofvarying forms and complexities, and building physical assembliesfor most of them. We evaluate our method in various aspects, in-cluding a comparison with four baselines and with designs createdby a LEGO ® expert. Our method took only 36 sec. to compute theairplane model shown in Figure 1, while more complex models canbe realized in just minutes. In contrast, it took the human expertclose to 1.5 hours to design a comparable model for airplane usingexisting software. Finally, we show how our method consumes mo-tion annotations and produces dynamic Technic models that exhibithinge-style rotations and embed gear systems; see Section 6. LEGO ® construction. The first work aimed at automatic construc-tion of LEGO ® models using regular bricks was by Gower et al. [1998].The problem was formulated as a combinatorial optimization tomaximize a goodness measure for LEGO ® structures. Some follow-up works include Petrovič [2001] using evolutionary algorithms,Winkler [2005] using beam search, Testuz et al. [2013] using a graph-based algorithm, Stephenson [2016] using a multi-phase approach,and Lee et al. [2018] using a genetic algorithm. Considering notonly the target shape, Luo et al. [2015] developed a comprehensivemethod for buildable LEGO ® structures in larger scales, consideringthe brick colors in the assemblies and the structural stability. Later,Yun et al. [2017] improved LEGO ® constructions by silhouette fitting.See [Kim et al. 2014] for a survey on LEGO ® layout methods.Besides regular brick models, Lambrecht [2006] computed LEGO ® assemblies with oriented thin plates. Waßmann and Weicker [2012]devised a two-phase approach for stability analysis by solving amax-flow network. Kuo et al. [2015] computed brick sculptures frompixel arts by considering visual quality and stability.So far, existing computational works have focused on regularLEGO ® bricks. In comparison, LEGO ® Technic designs involve a sig-nificantly more complex brick set and entirely different assemblymechanisms using a variety of pins and connectors; they even sup-port dynamic functionalities, such as joint rotations, that our cur-rent method is able to realize fully automatically. On the otherhand, while there are a number of commodity software tools to aidusers to design LEGO ® models, e.g., LEGO ® Digital designer [TheLEGO ® Group 2018a], MLCad[2018], and LDview [2018], such toolsonly provide basic modeling and rendering for users to create LEGO ® designs via simple drag-and-drop. We are not aware of any advancedcomputational support for designing LEGO ® Technic models.
ACM Trans. Graph., Vol. 38, No. 6, Article 196. Publication date: November 2019. omputational LEGO ® Technic Design • 196:3
Fig. 4. Overview of our approach. First, we construct a guiding graph (b) to abstract the user-input sketch (a), which has optional annotations for jointrotations. Then, we estimate the local orientation of beams (c), and iteratively refine a random initial layout to cover the input sketch, guided by the objectivefunction and constrained by the specified joint rotations (d). Further, we analyze the stability, balance, and assemblability of the generated model (e), andproduce assembly instructions (f) to facilitate physical construction (g). Upon the analysis, we may go back and revise the input sketch: (e) → (a). Assembly-based fabrication.
Lau et al. [2011] took a 3D man-madeobject as inputs, and generated parts and connectors for building theobject. Thomaszewski et al. [2014] designed motorized assemblies oflinkage-based characters. Schulz et al. [2014] proposed a data-drivenmethod to design 3D models. Cignoni et al. [2014] generated ribbon-shaped interlocking planar slices for assembling complex 3D shapes.Skouras et al. [2015] designed assemblies made up of interlockingquadrilateral elements. Song et al. [2016] built 3D-printed objects bycollectively using 3D-printed and laser-cut parts. Recently, Yao etal. [2017] created an interactive tool for designing decorative joints,while Geilinger et al. [2018] presented a design tool for robots thatmove using arbitrary arrangements of legs and wheels.In architecture modeling, Deuss et al. [2014] minimized the as-sembly work for building masonry structures; Yoshida et al. [2015]devised a method for building architecture-scale models formed byglued chopsticks; Pietroni et al. [2017] designed a computationalframework for tensegrity structures; and Desai et al. [2018] devel-oped a computational design system for electromechanical devices.Most previous works from this domain decompose target shapesinto customized parts to facilitate 3D fabrication. In contrast, ourgoal is to reconstruct a 3D shape using a diverse but fixed (thus notcustomized) brick set with a rich variety of brick connections, toapproximate the target shape specifications geometrically, struc-turally, and in terms of dynamics, while ensuring coherence of thebrick connections. Our problem is that of a multi-tiling rather thanshape decomposition. Instead of accounting for various constraintsrelated to 3D constructions, we must meet complex and multifacetedobjectives, including simplicity, symmetry, rigidity, etc., to supportthe Technic constructions. We are not aware of any previous com-putational assembly works that consider an assembly of multiplebrick types with diverse brick connections in 3D.
Application of LEGO ® bricks. Lastly, we discuss applications thatuse LEGO ® bricks as off-the-shelf building elements. In earlier works,Mitra and Pauly [2009] used LEGO ® bricks to build shadow art sculp-tures and Baronti et al. [2010] considered LEGO ® structures com-posed of regular bricks as markers for camera calibration. Song etal. [2012] used LEGO ® bricks with flat tiles to build polycube-shapedinterlocking puzzle pieces, while Mueller et al. [2014] developed the faBrickator system for rapidly prototyping functional objects by sub-stituting parts of the 3D objects with LEGO ® brick assemblies. Mostrecently, Chen et al. [2018] proposed to fabricate 3D objects with3D-printable pyramidal parts to form the outer shells and universalblocks (or LEGO ® bricks) to build the internal cores. Input and output.
We provide a GUI tool for sketching line seg-ments to craft 3D LEGO ® Technic designs, where the line segmentsare constrained to have integer lengths to match Technic bricksand nearby line segment endpoints are automatically snapped tojoin. Also, our tool shows semi-transparent guiding planes alignedwith the principal axes for sketching coplanar lines, since Tech-nic bricks mostly lay on the principal ( xy , yz , and zx ) planes; seeFigure 2(b). Optionally, the user may provide motion annotationsto specify hinge-style rotations at joints and to indicate embeddeddynamic parts in the sketch designs. The output of our tool includesa LEGO ® Technic design composed of Technic bricks, and assemblyinstructions and visualizations for the assembly process.
Objectives.
Our method aims to optimize a combination of thefollowing three objectives for the generated LEGO ® Technic designs: • Faithfulness to input line sketch.
First, the output designsshould resemble the input sketches and respect the intendedsymmetry and motion specifications provided by the user. • Structural integrity.
Second, the output should be coherently-connected and assemblable, while striving for rigidity. • Simplicity and efficiency.
Third, we strive for an output designthat is both simple and cost-effective for assembly.
Challenges.
The automatic generation of LEGO ® Technic modelsinvolves four sub-problems: (i) which bricks to use and where to putthem; this is analogous to the set cover problem, since the bricksshould cover the input design; (ii) beam/brick orientation in thelayout; (iii) layering (see Figure 3); and (iv) beam/brick connections.These four sub-problems are closely coupled, thus complicating thealgorithm design. Also, we have to attentively consider the objec-tives during the construction process, while efficiently exploring
ACM Trans. Graph., Vol. 38, No. 6, Article 196. Publication date: November 2019.
Fig. 5. LEGO ® Technic bricks with their unique brick IDs (left) and connection mechanisms (right) supported in our system. The connection mechanisms aregrouped by the pin head ratio ( ρ ), which is defined for quantifying the simplicity of connection mechanisms. In addition, we show as examples the abstractedline graphs for two of the connection mechanisms. See Supplementary material part A for more details on the brick set and connection mechanisms.Fig. 6. An example guiding graph (left) and edge cover (right). the immense search space for finding the optimal solution, i.e., alayout of bricks that form the input design. Our approach.
First, we study existing Technic models [Isogawa2010; Kmieć 2016], and enumerate the brick properties and connec-tion mechanisms. Then, we abstract each input as a guiding graph and each brick as a line graph to facilitate computation (see Fig-ures 4 (a) & (b)). Next, we formulate an objective to meet the variousconstruction requirements, develop the layout modification opera-tor to locally update brick layouts, and design a beam connectionprocedure to join adjacent beams. Further, we formulate the layoutsearch as a sketch cover to automate Technic constructions: • For simple connections and structural integrity, Technic beamsin local structures often have the same orientation. Hence, wefirst estimate the local orientation of beams over the designfor sub-problem (ii) in the first stage; see Figure 4(c). • Next, in the second stage, we optimize mainly for sub-problems(i), (iii) and (iv) altogether by first initializing a random layout,then formulating an iterative procedure to refine the layoutand connect beams, guided by the objectives; see Figure 4(d). ® TECHNIC CONSTRUCTION4.1 Technic Bricks
Brick set.
Figure 5 (left) shows the bricks supported by our sys-tem. For each brick, our system stores its 3D mesh model, physicalproperties such as weight, and connecting locations on the brick,i.e., holes on beams and connectors, and pin heads on pins and axles; see Figure 5 (right) for examples. Also, we abstract the structureof each brick as a simple line graph in 3D. For example, the greenbeam in Figure 5 (left) is abstracted as a graph of nine nodes (holes)and eight one-unit-long edges between adjacent nodes.
Brick connections.
Next, we enumerate the beam connection mech-anisms by studying the connection mechanisms in existing Technicmodels, where pin heads are locations that connect to beam holes,and connector bodies are non-pin-head locations in the connectionmechanisms; see the labels in Figure 5 (right). Also, we abstract eachmechanism as a line graph in 3D; see Figure 5 (right) for two exam-ples. We consider two types of pin heads, i.e., axle and regular , forconnecting respective types of beam holes;see the right inset figure. The two typestradeoff between connection rigidity andflexibility: the axle pin heads enforce rigidnon-rotatable connections that must beperpendicular/parallel, while the regularpin heads allow rotatable connections.
Pin-head ratio, ρ . In LEGO ® Technic, simpler connection mecha-nisms dominated by pin heads (not connector bodies) are preferred,since they help enhance the structural integrity. To quantify thesimplicity of connection mechanisms, we define ρ as the pin headcount over the total node count in a mechanism’s line graph, wheresimpler mechanisms have larger ρ values; see Figure 5 (right). Layer number, l ( b i ) . Technic beams of the same orientation areconnected in layers, above or below one another; see Figure 3. Givena beam, say b i , we define its layer number in a LEGO ® Technic con-struction as l ( b i ) , where l ( b i ) is zero, if b i exactly goes through theassociated sketch line in the input, and l ( b i ) is positive/negative, if b i is above/below the associated sketch line; see Figure 6 (right). Guiding graph.
For efficient computation, we abstract the inputsketch as a guiding graph (denoted as G ), where nodes are distributedalong the sketch line segments with adjacent nodes being one unitapart, like holes on Technic beams. Moreover, edges in G are all oneunit long for connecting the adjacent nodes; see Figure 6 (left). ACM Trans. Graph., Vol. 38, No. 6, Article 196. Publication date: November 2019. omputational LEGO ® Technic Design • 196:5
Fig. 7. Component terms in the objective function, from left to right: faithfulness to input sketch ( F f ), model simplicity ( F s ), and structural integrity ( F i ). Sketch cover.
An edge in G is said to be covered by a brick (e.g.,beam) if part of the brick is parallel to it, either crossing it or locatingat a small distance from the edge; see Figure 6 (right). If all edgesin G are covered by some bricks in a generated Technic model, themodel is said to fully cover the input sketch. The sketch cover problem is to find a brick layout that fully covers the given input design. Symmetric groups.
Our system considers reflection and transla-tional symmetry. We group line segments (i.e., subgraphs in guidinggraph) and mark symmetric groups in the input sketch. In our cur-rent implementation, this is done manually, but in the future, weplan to incorporate symmetry detection methods for the task.
We formulate and minimize the following objective function to guidethe finding of a beam layout (say B = { b i } ) that covers the inputsketch, while conforming to the various construction requirements: F = F f + F s + F i , where F f , F s , and F i are component terms in the objective (see thecorresponding illustrations shown in Figure 7): (i) Faithfulness to input sketch, F f . We define F f = w dev F dev + w cpt F cpt + w sym F sym , where w dev , w cpt , and w sym are weights, and we have: • F dev minimizes the distance deviation (i.e., layer number l ( b i ) )of the beams in the layout from the input sketch: F dev = (cid:115) (cid:205) b i ∈B L ( b i ) (cid:213) b i ∈B L ( b i ) l ( b i ) , where L ( b i ) denotes the length (number of holes) of beam b i . • F cpt compacts the layering by minimizing the range of l ( b i ) in each coplanar component (see Section 5.1) in the layout: F cpt = max C j ∈C [ max b i ∈C j l ( b i ) − min b i ∈C j l ( b i ) ] , where C = {C j } is a set of coplanar components extractedfrom the input design at the end of the stage one of our frame-work, and each C j is a subset of beams in B ; see Section 5.1for how we extract C , the set of coplanar components. • F sym minimizes the deviation from symmetry for each pairof symmetric beam groups: F sym = |S| (cid:213) (B i , B j , T ji )∈S (cid:205) b ∈B i L ( b ) d H (B i , T ji (B j )) , where B i ⊂ B and B j ⊂ B are symmetric groups; T ji is the3D symmetric transformation to bring B j to B i ; S is a set ofsymmetric groups; and d H denotes the Hausdorff distance. Inthe case of self (reflection) symmetry, we can have B i = B j . (ii) Model simplicity, F s . We encourage layout simplicity by mini-mizing the total beams length and maximizing the total pin headratios of the connection mechanisms in the layout; F s = w tbl (cid:205) b i ∈B L ( b i )|V| + w phr ( − ¯ ρ ) , where V is the set of nodes in guiding graph G ; ¯ ρ is the averagepin-head ratio (phr) over all the connections in the LEGO ® Technicmodel; and w tbl and w phr are weights. (iii) Structural integrity, F i . We define: F i = w col F col + w gap F gap + w coh F coh + w rgd F rgd , where w col , w gap , w coh , and w rgd are weights, and we have: • F col measures the total number of collisions ( N col ) betweenbeams in layout B and normalizes it by the total beam length: F col = (cid:205) b i ∈B L ( b i ) N col (B) . • F gap measures the total number of gaps between beams. Wedenote B v as the set of beams that associate with vertex v ∈ V . Then, we detect the gap at v bygap ( v ) = [ max b i ∈B v l ( b i ) − min b i ∈B v l ( b i ) ] + − |B v | , and define F gap = |V | (cid:205) v ∈V gap ( v ) . Although we mayfill a gap using a single hole brick (ID 18654 in Figure 5), butwe still need to reduce gaps for structural integrity. • F coh measures the connection coherence (connect-ability) be-tween beams in B . To start, we first connect adjacent beams in B by trying various connection mechanisms (see Section 5.2),and count the number of failure connections ( N cfail ). We de-fine F coh = N cfail |E | , where E is the edge set in guiding graph.Note that if we fail to connect two adjacent beams, the layer-ing and/or orientation of the beam(s) will be modified in theiterative refinement process; see Section 5.3 for detail. • F rgd measures the number of rigid beam subsets in the layout.In general, a local beam connection can be rigid or non-rigid;see Figure 7 (right). Here, we perform a depth first traversalover the guiding graph to examine the connections betweenadjacent beams, and stop the traversal at non-rigid connec-tions. As a result, we can decompose B into “rigid” subsets,such that all the beams inside are rigidly connected transi-tively. In this way, we can evaluate F rgd (B) = number of rigid subsets in B (cid:205) v ∈V ( deg ( v ) − ) , where deg ( v ) is the valence of vertex v in G . Note that wealso tried to normalize the term by using the number of beamsin B instead, but we find the above formulation to be more ACM Trans. Graph., Vol. 38, No. 6, Article 196. Publication date: November 2019.
Fig. 8. Optimizing the “beam hole” orientations, such that most adjacent beam holes have the same orientation for simpler beam connections. scalable to various input designs, since it measures not onlythe scale but also the topological complexity.To balance the components in the objective, we empirically set theassociated weights as follows: w cpt , w sym , w phr , w col , w gap , and w coh are set as 1, 30, 10, 30, 100, and 50, respectively. On the other hand, w dev , w tbl , and w rgd are set based on user preference. In practice,we set { w dev , w tbl , w rgd } as { , , } to aim for high faithfulnessto the input sketch, as { , , } to aim for model simplicity, andas { , , } or { , , } to additionally aim for rigid connec-tions. Figure 22 and Table 2 show experiments on their effectiveness. ® TECHNIC CONSTRUCTION SEARCH
In our initial attempts, we tried a greedy approach that progres-sively arranges locally-optimum beams to cover the sketch. Further,we tried several other approaches (see method comparisons in Sec-tion 6) to improve the search, but the results produced from theseapproaches are poorly optimized, especially for nontrivial inputs.To address the immense search space (combinations of beam place-ments, orientations, connections, and layering) in Technic construc-tions, the search has to be efficient and allow updates that iterativelypropagate over the layout. Hence, we design a two-stage approach(see Figures 4(c) & (d)) that first estimates the beam orientation theniteratively refines the layout to optimize the objectives.
In LEGO ® Technic models, adjacent beams of same orientation canbe steadily connected by pins. However, to build 3D models, we gen-erally need to arrange beams in different orientations for buildingdifferent parts of the models. Since the beam orientation stronglyaffects the overall structure, connections and joint rotations, we firstestimate the local beam orientation over the guiding graph.Mathematically, we represent a beam’sorientation as a 3D vector that passesthrough the medial axis of the beam’sholes; see the inset figure. In general, wecan reorient a beam, as long as its orienta-tion vector is perpendicular to either its corresponding line segmentin the sketch, or the edges that it covers in the guiding graph. Here,we model the problem of finding the beam orientation as an assign-ment problem. Since most nodes in guiding graph G will eventuallybe covered by beams in the generated LEGO ® Technic model (therest will be covered by the connection mechanisms), we create a holeorientation variable for each node in G . Since LEGO ® Technic bricksmostly lay on the principal (xy, yz and zx) planes, we should assign
Fig. 9. The optimized airplane in Figure 8 has 22 components. to each variable a principal direction (X, Y or Z), unless purposelyspecified in the user interface.
Goal. To minimize the number of adjacent node pairs with differenthole orientations , such that we can encourage the use of pin and axleconnections for model simplicity and structural integrity. Constraints. (i) For non-junction nodes along line segments inthe input sketch, their hole orientations should be perpendicularto the associated line segment ; (ii) orientation of adjacent holesshould be the same or perpendicular to each other, so that we mayuse a single beam to cover the two holes or connect them using aconnection mechanism shown in Figure 5 (right); (iii) at each jointannotated to allow rotation, the orientation of the associated holemust align with the joint’s rotational axis; and (iv) hole orientationvariables at symmetry locations are constrained to be the same (fortranslational symmetry) or mirrored (for reflection symmetry). Method.
We solve this combinatorial optimization problem usinga simulated annealing model [Kirkpatrick et al. 1983]. Initially, werandomize all hole orientation variables (see Figure 8 for examples)but following the listed constraints. Then, we iteratively choose arandom line segment, change the orientation of all the non-junctionnodes in the segment, and update the orientation at each associatedjunction node, if the objective is minimized. Figure 8 shows theinitial state, optimized result, and objective values for the threeexamples. In our implementation, we set the initial temperature as2 × , stopping temperature as 0 .
01, and cooling rate as 1 − |V | ,where V is the node set in the guiding graph. The optimizationcompletes in only a few seconds for most models; see Section 6. Find “components” and “beam placements”.
Next, we decomposenodes in guiding graph into connected and coplanar components,where the nodes in each component have the same orientation . For Except for short line segments with just a single interiorhole (see the right inset figure), since we observe that in sucha situation, existing LEGO ® models may fill/cover the nodeusing a single hole brick (near Figure 5 (top-left)) whoseorientation aligns with the associated line segment.ACM Trans. Graph., Vol. 38, No. 6, Article 196. Publication date: November 2019. omputational LEGO ® Technic Design • 196:7
Fig. 10. Nine possible beam placements to cover the red edge (see top left). example, the optimized cube and lifter shown in Figure 8 have sixand nine components, respectively, while the optimized airplanehas 22 components; see Figure 9. Also, we store symmetry informa-tion within and between components to facilitate later computation.Furthermore, on each component, we find all feasible beam place-ments on the component, but ignoring the beam placements thatpass through the annotated rotating joints. Hence, for each edge ina component, we keep a list of feasible beam placements that cancover the edge (see Figure 10 for an example), and also, a list ofbeam placements that stop at or pass through each node. Using thisdata structure, we can efficiently arrange beams to cover any edgein the guiding graph, and accelerate the layout generation in thesecond stage (layout modification operator) of our method.
Layout modification operator.
There are two key components inthe second stage of our method. The first one is an operator tomodify a given beam layout. We design this operator with the fol-lowing considerations: (i) the operator must be efficient , due to itsheavy usage in the search process; (ii) even if the operator is local,successively applying it should produce diverse beam layouts ; and(iii) it should avoid obviously bad beam placements .Figure 11 shows the operator procedure. After randomly picking acovered edge in the guiding graph, we remove all the beams that stopat or pass through the edge, and locate all possible beam placementsthat can cover the resulting “uncovered” edges in the guiding graph.To promote layout diversity, we next calculate a selection probabilityfor each possible beam placement, where each beam candidate isselected based on the number of uncovered edges that it can cover;see examples in Figure 11. Next, based on the probabilities, werandomly select a candidate beam placement to add to the layout,at a layer that produces more compact layering. Hence, we may tryboth long and short beams in the search, while avoiding meaninglessbeams that cannot cover any edge and encouraging model simplicityand faithfulness to the input sketch. We repeat this select-and-addprocess (usually a few times) until we cover all the uncovered edges.This procedure was carefully designed after experimenting withseveral alternatives; see Supplementary material part C.
Beam connection procedure.
Another key component is the proce-dure to connect adjacent beams, such that we can join the beamsand form a connected assembly in the end. Procedure-wise, given alayout of beams, we first identify edges in the guiding graph that arenot covered by any beam, and locally group the adjacent uncoverededges; see the examples shown in Figure 13 (leftmost column). Then,
Fig. 11. The layout modification operator efficiently modifies a layout bylocally removing beams around a random edge and adding new beams.
Not covered Covered by beam Covered by connections
Fig. 12. After we arrange the beams (left), we need to further arrangeconnection mechanisms (in orange) between adjacent beams (right).Fig. 13. Procedure: find connection mechanisms to join adjacent beams. we identify the beam holes around each group based on a distancethreshold of √ all feasible connection mechanisms that can join holes of differentbeams, then find subsets of them (see Figure 13 (rightmost column))that satisfy the following considerations: (i) the subset of mecha-nisms should together connect all the different beams around theuncovered edges; (ii) the chosen mechanisms should not collide withthe existing beams and also one another; (iii) we aim for maximal ACM Trans. Graph., Vol. 38, No. 6, Article 196. Publication date: November 2019. pin-head ratio for model simplicity; (iv) connecting bricks that ex-tend outward should not go below the ground plane or collide withany embedded dynamic element; and (v) the chosen mechanismsshould not interfere any annotated joint rotations.In general, beam connections are not always one-to-one, i.e., seethe case in Figure 13 (top). First, some mechanisms can join threeor more beams together, e.g., the L-shaped mechanism in Figure 13(middle) and the T-shaped one with ρ = 3 / Overall, our solution search in Stage two starts by initializing arandom layout of beams, then iteratively modifies it to improve theobjective function; see Figure 4(d). Particularly, the layout startswithout beam connections, so we have to find appropriate connec-tion mechanisms to join the beams during the search. We have triedvarious optimization frameworks to optimize the objective function(see Figure 19), and in the end, adopted a simulated annealing modelproposed by Cagan et al. [1998] to regulate the solution search.
Layout initialization.
We create the initial layout by repeating thefirst two steps in the layout modification operator, i.e., randomly pickan uncovered edge in the guiding graph and select a feasible beamplacement to add into the initial layout. However, we deliberatelyselect beam placements with equal probability to generate a morerandom initial layout (see Figure 4(d) for an example), since a morerandom layout helps the annealing process avoid local minima.
Overall procedure.
Algorithm 1 outlines the search procedure.There are four input parameters, T max , T mid , T min and r , which de-note the starting temperature, middle cutoff temperature, endingtemperature, and cooling rate, respectively. We empirically set T max as 2 × , T mid as 10, T min as 10 − , and r as 0 .
999 for simple modelsand as 0 . F ) without evaluating the collision and connectable terms forcomputational efficiency, then switching to the full version objective( F ) when the layout becomes stable. Also, since the layout modifi-cation operator is local, we actually update the objective functionvalue based on the local changes in the layout. This can boost thecomputational efficiency for evaluating the objective function. Discussion.
The beam orientations estimated in Stage one maynot always be perfect; we further allow the layout modificationoperator to try different valid beam orientations (see the constraintsin Section 5.1), when the layout becomes stable. See Figure 8 (right)for the estimated orientation of the pontoon beams on the bottom ofairplane; re-orientating them allows simpler connections (see Fig-ure 1) that minimize the objective. Besides the exponential annealingschedule we adopted in the search, we have tried other schedules:
ALGORITHM 1:
Overall procedure for iterative layout refinement
Data: T max , T min , T mid , r , and Guiding graph GB current = initialize_layout (G) // initialize the beam layout T = T max // initialize temperature T B best = B current // initialize the best layout F obj = F o // use approx. obj. func. while T > T min doif T < T mid then F obj = F // switch to full obj. func. end B new = modify (B current ) // layout modification op. ∆ = F obj (B new ) − F obj (B current ) if exp (− ∆ / T ) > rand ( , ) then B current = B new // accept the change if F obj (B current ) > F obj (B best ) then B best = B current // update the best layout endend T = T ∗ r // update temperature T endreturn B best Fig. 14. Example balance analysis (top) and stress plots (bottom). linear, logarithmic, optimum [Nourani and Andresen 1998], andthermodynamic [De Vicente et al. 2003]. However, we found noobvious improvements in results and running time. This is likelybecause the search space is discrete rather than continuous, wherethe feasible solutions are far from one another. Also, we have triedto extend Algorithm 1 to be a population-based search [Van Hen-tenryck and Vergados 2007] by finding N instead of one solutionsin each iteration and keeping the best K for generating candidatesin the next iteration (we set N = 20 and K = 4). However, the solu-tion quality improves only slightly but the running time increasessubstantially, so we kept N = K = Our tool provides further analysis on the generated Technic model:(i) self-balancing — check if the model’s center of gravity is well-supported [McGhee and Frank 1968; Prévost et al. 2013; Schneider
ACM Trans. Graph., Vol. 38, No. 6, Article 196. Publication date: November 2019. omputational LEGO ® Technic Design • 196:9
Fig. 15. A photograph showing the physical assemblies of most LEGO ® Technic models generated by our method. From left to right, we have print_box , airplane , long_bow , cube , house , crossbow , bird , bridge , flying_kite , claw , picker , table_lamp , lifter , chair_front , chair_side , little_ferris , glasses , seasaw , bicycle , and robot ; see Table 1 for the statistics (number of input sketch lines, bricks, etc.) about these generated models. and Eberly 2002] (see Figure 14 (top)); (ii) a visualization of the stressdistribution — script the model as input to the ANSYS R19.0 (Aca-demic) software (see Figure 14 (bottom)); and (iii) assemblability— test if all bricks can be iteratively removed from the assemblywithout collision. Based on the analysis, we can also generate an as-sembly sequence that respects the model symmetry, produce scriptsto render a model assembly video, and generate LEGO ® -style assem-bly instructions with the help of the LPub3D software [Sandy 2018].See Supplementary material part D for implementation details. We employed our method to design and generate a rich variety ofLEGO ® Technic models, as listed in Table 1. All models are automat-ically generated from input sketches on a MacBook Pro with a dual-core Intel i5 CPU and 8GB RAM. Figure 15 photographs the physicalassemblies of twenty of the computed models, while Figure 17 showsthe renderings of four remaining larger ones. Nine of them can per-form motion dynamics, e.g., bird, claw, and long_bow. These re-sults demonstrate that our method is able to generate LEGO ® Technicmodels of varying size, shape, structure, and functionality, fromsmall models, such as picker, flying_kite and lifter, with lessthan 100 bricks, to medium-sized models, such as long_bow, claw,and airplane, as well as to large models with over 400 bricks, suchas castle and print_box. Particularly, the results show coherentconnections between bricks and the preserved symmetry. In termsof shape and structure, our method can generate large planar andnearly-planar models like tower_ and castle, as well as 3Dstructures of varying complexity. Please refer to the supplementaryvideo for the input sketches and animated results. Timing performance.
The rightmost part in Table 1 reports therunning times of our method. Stage one takes only a few secondsto complete, except for a few very large models like space_station
Table 1. Statistics of our results: (i) the input sketch complexity shows thetotal number and total length of sketch lines, and the number of extractedcoplanar components (see Section 5.1)); (ii) statistics of the generated modelsinclude the number of beams, total number of bricks (beams & connectingbricks), and model’s physical size; and (iii) our method’s running times.
Input sketch complexity
Statistics of generated models
Running time (seconds)
Model lines total length bricks phy. size (cm ) stage stage total time12 96 6 20 44 7x7 x7 0.09 2.78 2.8737 138 19 36 60 18x6x18 0.28 4.22 4.5031 214 3 24 45 12x10x6 0.29 5.42 5.7117 82 6 11 33 6x6x12 0.11 5.71 5.8217 82 6 11 41 6x6x12 0.13 6.32 6.4526 112 6 22 60 22x4x6 0.18 5.83 6.0116 91 2 21 47 14x20x5 0.13 6.33 6.4631 180 2 29 51 30x6x15 0.25 6.37 6.6229 108 7 17 36 4x9x26 0.21 6.58 6.7924 200 8 20 60 14x9x17 0.43 6.49 6.9243 277 7 85 175 34x4x38 0.09 7.18 7.2733 116 6 29 70 22x4x10 0.32 7.64 7.9657 260 9 28 90 9x9x20 0.55 7.50 8.0551 268 18 52 119 23x25x17 0.34 8.79 9.1349 242 3 45 93 17x7x23 0.48 9.76 10.2444 179 7 28 82 36x28x7 0.23 11.15 11.38113 401 22 91 208 36x31x15 1.35 34.99 36.34180 780 34 150 378 20x18x23 4.12 53.46 57.58145 1579 9 267 495 72x75x14 6.78 102.03 108.81305 2102 45 477 1030 70x53x22 19.02 210.09 229.11166 1468 16 310 841 33x33x34 8.56 268.70 277.26382 2524 77 482 1080 25x25x68 27.04 298.34 325.38168 1402 1 223 458 4x87x150 3.16 345.97 349.13124 1812 3 196 514 14x107x33 8.34 360.00 368.34 and tower_ . As expected, Stage two takes up most of the pro-cessing time, since it needs to iteratively refine the layout and brick ACM Trans. Graph., Vol. 38, No. 6, Article 196. Publication date: November 2019.
Fig. 16. Robustness of our method to brick sets: cubes generated by our method using different brick sets, from full to a single-beam set.Fig. 17. Larger LEGO ® Technic models generated by our method. From leftto right, we have tower_ , space_station , castle , and tower_ .Fig. 18. Scalability test: cube and flying_kite in increasing sizes. connections. Overall, the whole method completes in only a fewminutes, even for very large models with a thousand bricks. Notealso that we sort the rows (models) in Table 1 by the total runningtime to reveal that the running time depends not only on the numberof bricks but also on the complexity of the input models. Scalability.
We examine the scalability of our method by generat-ing models for cube and flying_kite in increasing sizes. Figure 18shows the statistics of the results, where our method can efficientlygenerate models in varying scales within minutes, and the numberof beams and running time increase roughly linearly.
Robustness to brick set.
To explore our method’s robustness tovariations in brick set, we start with a full set of beams to generate cube with the goal of simple layouts. Then, we gradually take outthe most frequently-used beam from the set and re-generate cube,until the set is empty. From the results shown in Figure 16, we cansee that our method can produce coherent structures for all differentbrick sets. Importantly, beam arrangement is not a simple decisionthat greedily chooses the longest beam, but a global optimizationthat considers the overall structural coherency, simplicity, and sym-metry. See the second cube in Figure 16, our method can skillfullyarrange L-shaped beams to minimize the brick consumption; seeSupplementary material part E for more results.
Efficiency comparison to alternative methods.
To evaluate the effi-ciency of our method, we compare it with four different methodsthat are built upon our framework, and use them to generate Tech-nic models, specifically for minimal gap and minimal beam counts:(i) a random method, which starts with a random layout and appliesthe layout modification operator to iteratively refine the layout for50000 n times ( n is the input problem size to be described later); (ii)a greedy method, which progressively adds locally-optimum beamsthat cover the most portions of the uncovered sketch; (iii) a beamsearch method [Medress et al. 1977], which builds a three-layeredsearch tree of partial layouts as internal nodes and iteratively up-dates the layout with the local best choice using a beam search widthof 75; and (iv) an ant colony optimization method, which generalizesthe set cover model in [Ren et al. 2010] to handle beam layering.In the comparison, the input sketches we employed are 2D uni-form grids of n -by- n cells ( n =1..10); each cell is 4 × ® Technic models for these grids,especially the larger ones, is that we have to minimize both gaps andbeam counts, while considering beam layering ; hence, we cannotsimply use the longest beams, which will easily lead to gaps.Figure 19 shows the comparison results: the number of gapsand beams, as well as the running times, for different grid sizes( n ). Except for greedy and random, most methods can generategap-free layouts for n ≤
4, where simply using the longest beamscan effectively produce good solutions. Interestingly, the randommethod fails to find gap-free layouts even for n = , n gets larger,random and greedy start to produce layouts with lots of gaps, whileant colony, beam search, and our method can still produce gap-freelayouts for n = , , n = , ,
7, butit fails to avoid gaps, since the search space is not only immensebut also discrete rather than continuous. On the other hand, beamsearch can make good local decisions and avoids gaps better than
ACM Trans. Graph., Vol. 38, No. 6, Article 196. Publication date: November 2019. omputational LEGO ® Technic Design • 196:11
Fig. 19. Compare our method with four alternative methods (see legend) interms of the number of gaps and beams in use, as well as the running times.The input models are uniform square grids with increasing sizes ( n ).Fig. 20. Comparing manual designs with our method on generating LEGO ® Technic grids of 2 ×
2, 3 ×
3, and 4 × ant colony; however, it takes much longer time to run and fails tominimize beam counts for large n . In contrast, our method is able toproduce gap-free layouts even for large grids ( n = Manual designs.
To obtain a sense of how difficult it is for humansto design LEGO ® Technic models, we recruited ten participants (6males & 4 females, aged 22 to 25). Among them, four had experiencein building Technic models. Here, we followed the above comparisonexperiment and asked the participants to design Technic models for2D grids with minimal gaps and beam counts. However, consideringhuman building effort, we considered only n = , , n =
1, and taught the participants the meaning of gaps. Then, wegave each participant at most an hour to work on each model, andrecorded the resulting gap count, beam count, and time spent.
Fig. 21. Models designed by the LEGO ® Technic expert and by our method.
Figure 20 shows the results; see Supplementary material partF for the models. For the smallest grid, all participants could findthe optimal solution in a few minutes, while the two larger gridsare more challenging: no one was able to find better or even thesame solutions as our method, which are gap-free with the minimalbeam count. For the largest grid, n =
4, only five participants foundgap-free solutions, while one of them managed to find a solutionwith 24 beams, but it was still not as good as the solution producedby our method. Note that before this experiment on 2D grids, we dida pilot study asking the participants to build LEGO ® Technic modelsof 3D cubes. However, those who did not have prior experiencefailed to connect bricks into cubes; some of them tried it for 50+minutes. We concluded that it would be too challenging for non-expert participants to arrange beams in 3D without gaps.
Compare with human expert.
Further, we recruited an expertwho had over five-year full-time working experience specialized inLEGO ® Technic design. In this experiment, we first showed to himsome input sketches without showing him the models generatedby our method, then asked him to design models that are faithfulto these inputs with the least number of bricks. His first commentwas that such requirement is the same as what he did in his dailydesigns. Also, we learnt that he preferred to create his designs us-ing the LEGO ® Digital Designer software [2018a] instead of manualassembly, since the software tool provides fast brick retrieval, morebrick choices, and can replicate symmetric sub-structures. Here, werecorded his design time and the number of bricks in use, and com-puted the beam deviations in the results. From the results shownin Figure 21, we can see that the LEGO ® Technic models producedby our method are similar to those designed by the expert, in termsof beam count, beam deviation, and visual appearance. However,designing the models using conventional software took the expertminutes to more than an hour, while our method is able to designcomparable solutions in less than a minute.
ACM Trans. Graph., Vol. 38, No. 6, Article 196. Publication date: November 2019.
Fig. 22. LEGO ® Technic models generated with emphasis on different termsin our objective, e.g., simplicity, rigidity, and similarity. Each row showsdifferent results from the same input sketch.
Designs created by participants using our tool.
We recruited sevenparticipants (5 males & 2 females, aged 21 to 24) to try our tool.Among them, two had experience in building LEGO ® Technic mod-els and one had drawing background. When they came to the lab, wetaught them our GUI and the requirements on the input sketch, thenshowed them a few example input sketches and the output Technicmodels. Then, each participant used around 15 min. to design whatthey wanted to create, and used around 37 min. on average to sketchtheir designs on our GUI. Figure 15 (bottom right) shows four of themodels designed and assembled by the participants, featuring moviecharacters, everyday objects, and abstract models: little_ferris(design: 12 min., generation: 6 sec.), bicycle (design: 17 min., gen-eration: 7 sec.), glasses (design: 27 min., generation: 10 sec.), androbot (design: 87 min., generation: 58 sec.).
Adapting to different objectives.
Our method can generate modelsfrom the same input sketch by emphasizing F dev , F tbl , and F rgd in our objective to different extent; see Section 4.3 for the specificweight setting. For example, we can aim for simple layouts withminimized brick count or aim for high input similarity at the expenseof using more bricks; see results for cube and airplane in Figure 22.Additionally, we can aim for connection rigidity and encourage ourmethod to connect adjacent beams with multiple holes; see cube(rightmost), long_bow, and tower_ in Figure 22. Ablation study on objective terms.
Further, we conduct an experi-ment to show how other terms in the objective function affect theresults. Here, we fix { w dev , w tbl , w rgd } as { , , } to ensure model Table 2. Effects of adjusting the weights of objective terms on number ofbeams ( |B | ), layer compactness ( F cpt ), model symmetry ( F sym ), averagepin-head ratio ( ¯ ρ ), number of collisions ( N col ), number of gaps ( N gap ), andnumber of failure connections ( N cfail ). We generate each result by indepen-dently modifying each weight, w cpt , w sym , w phr , w col , w gap , or w coh , whilefixing the others. The adjustment either nullifies (null) the effect of theassociated term by setting the weight to zero or emphasizes the effect bydoubling the weight (x2). Compared with the result (last column) generatedunder the default weights, we highlight the improved aspects in light greenand worsened aspects in light pink, showing that deviations from the defaultsettings may improve certain aspects but could worsen others.Fig. 23. Some of the generated models for flying_kite in Table 2. simplicity, then test each of the other objective terms by indepen-dently adjusting its associated weight. We set the weight to zeroto assess the necessity of an objective term, or double its value toincrease its impact. Table 2 summarizes the ablation study resultson two input models. Without changing the annealing temperaturesand cooling rate, we found that doubling any particular weightusually breaks the balance among the objective terms and coulddeteriorate the results. On the other hand, neglecting a particularterm could deteriorate the corresponding aspect in the resultingmodel, without improving much on the other aspects. Figure 23shows some of the models generated in this ablation study. Dynamic models.
Figure 24 shows physical assemblies designedwith annotated joint rotations. Comparing the two chairs, we cansee that by annotating joints with different rotation axes, our methodcan constrain the connecting beams to rotate at specific orientationsat the joints, thus leading to the production of different models.Besides, given a mechanical system (see the two examples in Fig-ure 25), we can force the generated beams in the layout to exactlypass through some annotated sketch lines, such that we can thenembed a mechanical system into the generated model. Please seethe supplementary video for these models in action.
ACM Trans. Graph., Vol. 38, No. 6, Article 196. Publication date: November 2019. omputational LEGO ® Technic Design • 196:13
Fig. 24. Example models with annotated joint rotations generated from sketches. From left to right,we have chair_front (top left), chair_side (bottom left), table_lamp , seasaw , and claw . Fig. 25. Top: house directly generated from the in-put sketch. Middle: beams are constrained to exactlypass through the annotated sketch line (in blue) forembedding a gear system. Bottom: long_bow em-bedded with a shooting mechanic. Discussion on global rigidity.
If we optimize the long_bow modelwithout the rigidity term, the model could be deformed due to thegravity, as demonstrated in Figure 22 (middle left). To evaluate theglobal rigidity of a LEGO ® Technic structure is a very challengingproblem. First, global non-rigidity is a result of multiple (a subset of)joints in the overall structure. Particularly, a joint may be transitively(or indirectly) blocked to rotate by others that are not located nextto it. Here, trying every joint subset would require tremendouscomputation. Also, we have to deal with a diverse and irregularbrick set that can be connected in many different ways. Further,we need a unified model to evaluate the effectiveness of individualconnections, brick-blocking relations, and other physical constraintssuch as friction. Hence, we believe analytically evaluating the globalrigidity is very challenging, and will require extensive works.In the course of this work, we have thought of two approaches tothis problem: (i) apply external forces on the structure, then see ifthe structure deforms in a physical simulation; and (ii) formulate amotion-based equation by setting velocity variables on every joint,constrain them based on the beam connections, solve it, and locatethe joints with non-zero velocities. Clearly, these approaches arepreliminary and require more thoughtful ideas to turn them intosolid feasible solutions. Hence, we only consider local rigidity inthis work, and leave global rigidity as our future work.
We presented a first attempt to computerize LEGO ® Technic con-structions. Altogether, there are three contributions in this work.First is an automatic computational method that can efficiently gen-erate LEGO ® Technic models from user input sketches. Particularly,the generated model is a coherently-connected structure composedof LEGO ® Technic bricks, and we can aim for faithfulness to inputsketch, model simplicity, and structural integrity in the model gen-eration. Second is our comprehensive model for various aspects inLEGO ® Technic constructions, including the enumeration of brickproperties and connection mechanisms, conceptualization of theinput sketch as a guiding graph, formulation of the constructionrequirements into an objective, and dynamic model constructionswith hinge-style rotations and dynamic parts embedding. Third, we
Fig. 26. Failure cases from our current computational method: intersectingsketch lines (left) and closely-packed sketch lines (right). also developed a working system to sketch the inputs, and to analyzethe balance, stress, and assemblability of the generated model. Inthe end, we employed our system to create LEGO ® Technic construc-tions of various shapes, complexities and functionalities, comparedit with four alternative methods, general users and a human expert,evaluated it for scalability, robustness and adaptiveness, as well asphysically built most of the generated models.
Limitations.
In terms of model generation, while our methodmakes an effort to adjust the beams, it may still fail to create con-nections in some situations, especially when several beams inter-sect/touch one another non-orthogonally; see Figure 26 (left). Fordense and parallel sketch lines, the generated models may not retainthe gaps between the lines; see Figure 26 (right). Also, our currentmethod assumes most sketch lines are covered by the beam bricks.However, as shown in the expert’s designs, axles may also be usedto cover the sketch lines, where some special axle-related connec-tor bricks can be used in the connections. Moreover, as discussedearlier, our current formulation for connection rigidity is local, notglobal. Lastly, in our sketching tool, diagonal lines in the sketchesshould follow certain Pythagorean triples, e.g., after we sketch anL-structure with two orthogonal lines, if we want to use an extraline to diagonally connect the previous two lines, the length of thediagonal line may be five-unit long, since 5 = + .In terms of our handling of dynamic constraints in the input,while our method realizes user-annotated hinge-style rotations, thedecision of where to put joints and how to make different partswork together to realize a desirable dynamic behavior still remainshard for novice users. Also, besides hinge-style rotations, otherLEGO ® Technic motions such as sliding, sheering, lifting, and theircombinations are not yet considered in our system.
ACM Trans. Graph., Vol. 38, No. 6, Article 196. Publication date: November 2019.
Discussion and future work.
Addressing the various limitationsabove already suggests a comprehensive LEGO ® Technic design sys-tem involving a number of sub-problems, such as global rigidityanalysis, rigid structure generation, inverse joints computation, andinverse multi-functionality design with mechanical elements. More-over, official LEGO ® Technic models often contain customized partsspecially-designed for the outer shell of the model. We would alsolike to design and fabricate customized 3D-printed parts to workwith the beams and connectors in the LEGO ® Technic system fordriving 3D-printed customized models.
ACKNOWLEDGMENTS
We thank all the anonymous reviewers for their comments andfeedback. We also acknowledge help from Ruihui Li for UI develop-ment, Shufang Wang and Tianwen Fu for model assembly, Chun YuLiu for designing the models shown in Figure 21, and Wallace Liraand Johannes Merz for paper proofreading. Figures 2 (a), (b) and (c)are courtesy of YouTuber TECHNICally Possible, Nicolas Lespour,and Will Gorman, respectively. This work is supported in part bygrants from the Research Grants Council of the Hong Kong SpecialAdministrative Region (Project no. CUHK 14201918 and 14203416),NSERC grants (No. 611370), and gift funds from Adobe.
REFERENCES
Luca Baronti, Matteo Dellepiane, and Roberto Scopigno. 2010. Using Lego Pieces forCamera Calibration: a Preliminary Study. In
Eurographics (short paper) . 97–100.Jonathan Cagan, Drew Degentesh, and Su Yin. 1998. A simulated annealing-basedalgorithm using hierarchical models for general three-dimensional componentlayout.
Computer-aided design
30, 10 (1998), 781–790.Xuelin Chen, Honghua Li, Chi-Wing Fu, Hao(Richard) Zhang, Daniel Cohen-Or, andBaoquan Chen. 2018. 3D Fabrication with Universal Building Blocks and PyramidalShells.
ACM Trans. on Graph. (SIGGRAPH Asia)
37, 6 (2018).Paolo Cignoni, Nico Pietroni, Luigi Malomo, and Roberto Scopigno. 2014. Field-alignedMesh Joinery.
ACM Trans. on Graph. (SIGGRAPH Asia)
33, 1 (2014). Article no. 11.Travis Cobbs and Peter Bartfai. 2018. LDview. [Online; accessed18-May-2018].Juan De Vicente, Juan Lanchares, and Román Hermida. 2003. Placement by thermody-namic simulated annealing.
Physics Letters A
The 31st Annual ACM Symposium on User InterfaceSoftware and Technology . ACM, 457–472.Mario Deuss, Daniele Panozzo, Emily Whiting, Yang Liu, Philippe Block, Olga Sorkine-Hornung, and Mark Pauly. 2014. Assembling Self-supporting Structures.
ACM Trans.on Graph. (SIGGRAPH Asia)
33, 6 (2014). Article no. 214.Moritz Geilinger, Roi Poranne, Ruta Desai, Bernhard Thomaszewski, and Stelian Coros.2018. Skaterbots: optimization-based design and motion synthesis for roboticcreatures with legs and wheels.
ACM Trans. on Graph. (SIGGRAPH)
37, 4 (2018).Rebecca A. H. Gower, Agnes E. Heydtmann, and Henrik G. Petersen. 1998. LEGO:Automated Model Construction. In
European Study Group with Industry . 81–94.Yoshihito Isogawa. 2010.
LEGO ® Technic Idea Book: Fantastic Contraptions . No StarchPress.Jae Woo Kim, Kyung Kyu Kang, and Ji Hyoung Lee. 2014. Survey on Automated LEGOAssembly Construction. In
Proc. WSCG . 89–96.Scott Kirkpatrick, C. Daniel Gelatt, and Mario P. Vecchi. 1983. Optimization by simulatedannealing.
Science
The Unofficial LEGO ® Technic Builder’s Guide, 2nd Edition .No Starch Press.Ming-Hsun Kuo, You-En Lin, Hung-Kuo Chu, Ruen-Rone Lee, and Yong-Liang Yang.2015. PIXEL2BRICK: Constructing Brick Sculptures from Pixel Art.
ComputerGraphics Forum (Pacific graphics)
34, 7 (2015), 339–348.Michael Lachmann. 2018. MLCad. http://mlcad.lm-software.com/ [Online; accessed5-Nov-2018].Bram Lambrecht. 2006. Voxelization of boundary representations using orientedLEGO ® plates. University of California, Berkeley, http://lego.bldesign.org/ [Online;accessed 18-May-2018].Manfred Lau, Akira Ohgawara, Jun Mitani, and Takeo Igarashi. 2011. Converting3D Furniture Models to Fabricatable Parts and Connectors. ACM Trans. on Graph. (SIGGRAPH)
30, 4 (2011). Article no. 85.Seung-Mok Lee, Jae Woo Kim, and Hyun Myung. 2018. Split-and-Merge-Based GeneticAlgorithm (SM-GA) for LEGO Brick Sculpture Optimization.
IEEE Access
ACMTrans. on Graph. (SIGGRAPH Asia)
34, 6 (2015). Article no. 222.Robert B. McGhee and Andrew A. Frank. 1968. On the stability properties of quadrupedcreeping gaits.
Mathematical Biosciences
ArtificialIntelligence
9, 3 (1977), 307–316.Niloy J. Mitra and Mark Pauly. 2009. Shadow Art.
ACM Trans. on Graph. (SIGGRAPHAsia)
28, 5 (2009). Article No. 156.Stefanie Mueller, Tobias Mohr, Kerstin Guenther, Johannes Frohnhofen, and PatrickBaudisch. 2014. faBrickation: Fast 3D Printing of Functional Objects by IntegratingConstruction Kit Building Blocks. In
CHI . 3827–3834.Yaghout Nourani and Bjarne Andresen. 1998. A comparison of simulated annealingcooling strategies.
Journal of Physics A: Mathematical and General
31, 41 (1998),8373.Pavel Petrovič. 2001. Solving LEGO Brick Layout Problem using Evolutionary Algo-rithms. In
Proc. NIK (Norsk Informatikkonferanse) . 87–97.Nico Pietroni, Marco Tarini, Amir Vaxman, Daniele Panozzo, and Paolo Cignoni. 2017.Position-based Tensegrity Design.
ACM Trans. on Graph. (SIGGRAPH Asia)
36, 6(2017). Article no. 172.Romain Prévost, Emily Whiting, Sylvain Lefebvre, and Olga Sorkine-Hornung. 2013.Make It Stand: Balancing Shapes for 3D Fabrication.
ACM Trans. on Graph. (SIG-GRAPH)
32, 4 (2013). Article no. 81.Zhi-Gang Ren, Zu-Ren Feng, Liang-Jun Ke, and Zhao-Jun Zhang. 2010. New ideasfor applying ant colony optimization to the set covering problem.
Computers &Industrial Engineering
58, 4 (2010), 774–784.Trevor Sandy. 2018. LPub3D. https://trevorsandy.github.io/lpub3d/ [Online; accessed29-Dec-2018].Philip Schneider and David H. Eberly. 2002.
Geometric Tools for Computer Graphics .Morgan Kaufmann Publishers.Adriana Schulz, Ariel Shamir, David I.W. Levin, Pitchaya Sitthi-Amorn, and WojciechMatusik. 2014. Design and fabrication by example.
ACM Trans. on Graph. (SIG-GRAPH)
33, 4 (2014). Article no. 62.Mélina Skouras, Stelian Coros, Eitan Grinspun, and Bernhard Thomaszewski. 2015.Interactive Surface Design with Interlocking Elements.
ACM Trans. on Graph.(SIGGRAPH Asia)
34, 6 (2015). Article no. 224.Peng Song, Bailin Deng, Ziqi Wang, Zhichao Dong, Wei Li, Chi-Wing Fu, and LigangLiu. 2016. CofiFab: Coarse-to-fine Fabrication of Large 3D Objects.
ACM Trans. onGraph. (SIGGRAPH)
35, 4 (2016). Article no. 45.Peng Song, Chi-Wing Fu, and Daniel Cohen-Or. 2012. Recursive Interlocking Puzzles.
ACM Trans. on Graph. (SIGGRAPH Asia)
31, 6 (2012). Article no. 128.Ben Stephenson. 2016. A Multi-Phase Search Approach to the LEGO ConstructionProblem. In
Proc. Symposium on Combinatorial Search (SoCS) . 89–97.Romain Testuz, Yuliy Schwartzburg, and Mark Pauly. 2013. Automatic Generation ofConstructible Brick Sculptures. In
Eurographics (short paper) . 81–84.The LEGO ® Group. 2018a. LEGO ® Digital Designer, version 4.3.11. [Online; accessed 10-January-2018].The LEGO ® Group. 2018b. LEGO ® Technic official site. [Online; accessed 18-May-2018].Bernhard Thomaszewski, Stelian Coros, Damien Gauge, Vittorio Megaro, Eitan Grin-spun, and Markus Gross. 2014. Computational Design of Linkage-based Characters.
ACM Trans. on Graph. (SIGGRAPH)
33, 4 (2014). Article no. 64.Pascal Van Hentenryck and Yannis Vergados. 2007. Population-based simulated anneal-ing for traveling tournaments. In
Proceedings of the National Conference on ArtificialIntelligence , Vol. 22. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MITPress; 1999, 267.Martin Waßmann and Karsten Weicker. 2012. Maximum Flow Networks for StabilityAnalysis of LEGO ® Structures. In
Proc. Annual European Conference on Algorithms(Lecture Notes in Computer Science, vol 7501) . 813–824.David V. Winkler. 2005. Automated Brick Layout. In
Proc. BrickFest . 145–166.Jiaxian Yao, Danny M. Kaufman, Yotam Gingold, and Maneesh Agrawala. 2017. Interac-tive Design and Stability Analysis of Decorative Joinery for Furniture.
ACM Trans.on Graph. (SIGGRAPH)
36, 2 (2017). Article no. 20.Hironori Yoshida, Takeo Igarashi, Yusuke Obuchi, Yosuke Takami, Jun Sato, MikaAraki, Masaaki Miki, Kosuke Nagata, Kazuhide Sakai, and Syunsuke Igarashi. 2015.Architecture-scale Human-assisted Additive Manufacturing.
ACM Trans. on Graph.(SIGGRAPH)
34, 4 (2015). Article no. 88.Grim Yun, Cheolseong Park, Heekyung Yang, and Kyungha Min. 2017. Legorizationwith multi-height bricks from silhouette-fitted voxelization. In