Globally optimal point set registration by joint symmetry plane fitting
GGlobally optimal point set registration by joint symmetry plane fitting
Lan Hu, Haomin Shi, and Laurent KneipMobile Perception Laboratory, ShanghaiTech University { hulan, shihm, lkneip } @shanghaitech.edu.cn Abstract
The present work proposes a solution to the challengingproblem of registering two partial point sets of the same ob-ject with very limited overlap. We leverage the fact that mostobjects found in man-made environments contain a planeof symmetry. By reflecting the points of each set with re-spect to the plane of symmetry, we can largely increase theoverlap between the sets and therefore boost the registra-tion process. However, prior knowledge about the plane ofsymmetry is generally unavailable or at least very hard tofind, especially with limited partial views, and finding thisplane could strongly benefit from a prior alignment of thepartial point sets. We solve this chicken-and-egg problem byjointly optimizing the relative pose and symmetry plane pa-rameters, and notably do so under global optimality by em-ploying the branch-and-bound (BnB) paradigm. Our resultsdemonstrate a great improvement over the current state-of-the-art in globally optimal point set registration for commonobjects. We furthermore show an interesting application ofour method to dense 3D reconstruction of scenes with repet-itive objects.
1. Introduction
The alignment of two point sets is a fundamental geo-metric problem that occurs in many computer vision androbotics applications. In computer vision, the technique isused to stitch together partial 3D reconstructions in orderto form a more complete model of an object or environ-ment [23]. In robotics, point set registration is an essentialingredient to simultaneous localization and mapping withaffordable consumer depth cameras [20] or powerful 3Dlaser range scanners [5]. The general approach for aligningtwo point sets does not require initial correspondences. It isgiven by the Iterative Closest Point (ICP) algorithm [28], alocal search strategy that alternates between geometric cor-respondence establishment (i.e. by simple nearest neigh-bour search) and Procrustes alignment. The iterative proce-dure depends on a sufficiently accurate initial guess aboutthe relative transformation (e.g. an identity transformation in the case of incremental ego-motion estimation).The present work is motivated by a common problemthat occurs when performing a dense reconstruction of anenvironment which contains multiple instances of the sameobject. An example of the latter is given by a room inwhich the same type of chair occurs more than once. Letus assume that the front-end of our reconstruction frame-work encompasses semantic recognition capabilities whichare used to segment out partial point sets of objects of thesame class and type [14]. There is a general interest in align-ing those partial object point sets towards exploiting theirmutual information and completing or even improving thereconstruction of each instance. The difficulty of this partialpoint set registration problem arises from two factors: • The relative pose between the different objects is arbi-trary and unknown. • Since the objects are observed under an arbitrary poseand with potential occlusions, the measured partialpoint sets potentially have very little overlap.Our contribution focuses on the registration of only twopartial point sets, which appears as a worthwhile startingpoint given that the registration of more than two point setsmay be broken down into many pair-wise alignments. Theplain ICP algorithm is only a local search strategy that de-pends on a sufficiently accurate initial guess, which is whyit may not serve as a valid solution to our problem. A poten-tial remedy is given by the globally optimal ICP algorithmpresented by Yang et al. [26]. However, the algorithm stilldepends on sufficient overlap in the partial point sets, whichis not necessarily a given (50% is reported as a requirementfor high success rate).The core idea of the present work consists of exploit-ing the fact that the majority of commonly observed objectscontain a plane of symmetry. By reflecting the points ofeach partial point set with respect to the plane of symmetry,we may effectively increase the overlap between the twosets and vastly improve the success rate of the registrationprocess. However, given that each point set only observespart of the object, the identification of the plane of symme-try in each individual point set appears to be an equally diffi-1 a r X i v : . [ c s . C G ] F e b ult and ill-posed problem than the partial point set registra-tion problem itself. It is only after the aligning transforma-tion is found that symmetry plane detection would becomea less challenging problem. In conclusion, the solution ofeach problem strongly depends on a prior solution to theother. We therefore present the following contributions: • We solve this chicken-and-egg problem by a joint solu-tion of the aligning transformation as well as the sym-metry plane parameters. To the best of our knowledge,our work is the first to address those two problemsjointly, thus leading to a vast improvement over theexisting state-of-the-art in globally optimal point setregistration, especially in situations in which two pointsets contain very limited overlap. • We immediately provide a globally optimal solutionto this problem by employing the branch-and-boundoptimisation paradigm. Our work implicitly providesthe first solution to globally optimal symmetry planeestimation in a single point set, or—more generally—symmetry plane detection across two point sets. • We show a meaningful application of our algorithm ina dense 3D reconstruction scenario in which multipleinstances of the same object occur.
2. Related Work
Despite strong mutual dependency, our method is thefirst to perform joint point set alignment and symmetryplane estimation. Our literature review therefore cites priorart on those two topics individually.
The Iterative Closest Point (ICP) [6, 3, 28] algorithm isa popular method for aligning two point sets. It does notdepend on a prior derivation of point-to-point correspon-dences, and simply aligns the two sets by iteratively alter-nating between the two steps of finding nearest neighbours(e.g. by minimising point-to-point distances), and comput-ing the alignment (e.g. using Arun’s method [2]). To im-prove robustness of the algorithm against occlusions andreduced overlap, the method has been extended by outlierrejection [28, 12] or data trimming [7] techniques. How-ever, the classical ICP algorithm remains a local search al-gorithm for which the convergence depends on initial guessand sufficient overlap between both point sets.An entire family of alternative approaches relies on theidea of expressing both point sets by a Gaussian MixtureModel (GMM) and aligning the latter using Gaussian Mix-ture Alignment (GMA). Notable GMA-based techniquesfor rigid and non-rigid registration are given by the robustpoint matching algorithm by Chui and Rangarajan [8], thecoherent point drift strategy by Myronenko and Song [19], kernel correlation by Tsin and Kanade [24], and the GMM-Reg algorithm by Jian and Vemuri [15]. While GMA is ad-vertised by improved robustness against poor initialisations,noise, and outliers, another key advantage with respect topoint-based methods is given by a closed-form expressionto evaluate the quality of the alignment (i.e. the methoddoes not depend on an alternating search for nearest neigh-bours). However, the listed GMA algorithms remain localsearch algorithms, which makes them inapplicable in sce-narios in which no prior about the relative transformation isgiven upfront.In contrast, globally optimal algorithms avoid local min-ima by searching the entire space of relative transforma-tions, often using the branch-and-bound paradigm [17, 21,22]. Yang et al. [27, 26] propose the Go-ICP algorithm,which applies branch-and-bound to the ICP problem to findthe globally optimal minimum of the sum of L2-distancesbetween nearest neighbours from two aligned point sets.The method is accelerated by using local ICP in the loop.However, missing robustness of the cost-function causesthe method to remain sensitive with respect to occlusionsand partial overlaps. Campbell et al. [4] finally deviseGOGMA, a branch-and-bound variant in which the objec-tive of minimising point-to-point errors is again replaced bythe convolution of GMMs.Inspired by those recent advances, we also employ thenested branch-and-bound strategy integrated with local ICPto find a globally optimal alignment. However, in contrastto all prior art, we are the first to jointly fit a plane of sym-metry, which leads to a large improvement in partial scanalignment for common symmetric objects.
For a comprehensive review of symmetry detection, wekindly refer the reader to Liu et al.’s review [18]. Here weonly focus on the problem of symmetry plane fitting withmissing data. The most straightforward solution is given byemploying the RANdom SAmple Consensus (RANSAC)algorithm proposed by Fischler and Bolles [11], a well-known algorithm for robust model fitting for outlier affecteddata. In the context of shape matching, the basic idea is toextract sparse characteristic points and match them betweenboth sets. We then choose a random subset of correspon-dences and derive a hypothesis for the global transformationinduced by these samples. The alignment quality is finallyevaluated by the matching error between the two shapes.The method can be easily applied for detecting a plane ofsymmetry in a single point set by hypothesising the latterto be orthogonal to the axis connecting a correspondence.For example, Cohen et al. [10] detect symmetries in sparsepoint clouds by using appearance-based 3D-3D point cor-respondences in a RANSAC scheme. The detected symme-tries are subsequently explored to eliminate noise from the2oint-clouds. Xu et al. [25] present a voting algorithm todetect the intrinsic reflectional symmetry axis. Using theaxis as a hint, a completion algorithm for missing geometryis again shown. Jiang et al. [16] on the other hand proposean algorithm to find intrinsic symmetries in point clouds byusing a curve skeleton. A set of filters then produces a can-didate set of symmetric correspondences which are finallyverified via spectral analysis. Although these works showresults on partial data, the amount of missing data is typi-cally small. Inspired by the work of [9] which detects sym-metry by registration, we propose to detect the symmetryplane alongside partial point set registration, thus leading toimproved performance in situations with limited overlap.
3. Preliminaries
We start by introducing the notation used throughout thepaper and review the basic formulation of the ICP problemas well as planar reflections.
Let us denote the two partial object point sets by X = { x i } Mi =1 and Y = { y i } Ni =1 (sometimes called the model and data point sets, respectively). The goal pursued in this pa-per is the identification of a Euclidean transformation givenby the rotation R and the translation t that transforms thepoints of Y such that they align with the points of X . If X and Y contain points in the 2D plane, R and t form anelement of the group SE (2) . If X and Y contain 3D points, R and t will be an element of SE (3) . Note that alignment denotes a more general idea rather than just the minimisa-tion of the sum of distances between each point of Y andits closest point within X . The point sets have different car-dinality and potentially observe very different parts of theobject with only very little overlap. This motivates our ap-proach that takes object symmetries into account. The standard solution to the point set registration prob-lem is given by the ICP algorithm [28], which minimizesthe alignment error given by E ( R , t ) = N (cid:88) i =1 e ri ( R , t | y i ) = N (cid:88) i =1 (cid:107) Ry i + t − x j (cid:107) , (1)where e ri ( R , t | y i ) is the per-point residual error for y i , and x j is the closest point to y i within X , i.e. x j = argmin x ∈X (cid:107) Ry i + t − x (cid:107) . (2)Given an initial transformation R and t , the ICP algorithmiteratively solves the above minimization problem by alter-nating between updating the aligning transformation with fixed x j (i.e. using (1)), and updating the closest-pointmatches x j themselves using (2). It is intuitively clear thatthe ICP algorithm only convergence to a local minimum. Symmetry is modelled by a reflection by the symmetryplane. Let us define the symmetry plane by the normal n and depth-of-plane d such that any point x on the plane sat-isfies the relation x T n + d = 0 . Let x now be a (2D or 3D)point from X . The reflection plane reflects x to a singlereflected point ˆ x given by ˆ x = x − n ( x T n + d ) . (3)The term in parentheses is the signed distance between x and the reflection plane. The subtraction of n times thisdistance reflects the point to the other side of the plane. Theproblem of symmetry identification may now be formulatedas a minimisation of the symmetry distance defined by E ( n , d ) = M (cid:88) i =1 e si ( n , d | x i ) = M (cid:88) i =1 (cid:107) ˆ x i − x j (cid:107) , (4)where e si ( n , d | x i ) is the per-point residual error for x i , and x j is the nearest point to ˆ x i in X , i.e. x j = argmin x ∈X (cid:107) ˆ x i − x (cid:107) . (5)It is intuitively clear that the symmetry plane fitting problemmay also be solved via ICP, the only difference being theparameters over which the problem is solved (i.e. n and d ). Let us still assume that n and d are the symmetry planeparameters of a point set X , and R and t are the parametersthat align a point set Y with X . Each transformed point Ry + t and its transformed, symmetric equivalent R ˆ y + t must still fulfill the original reflection equation (3): R ˆ y + t = Ry + t − n (( Ry + t ) T n + d ) . (6)Cancelling t and multiplying by R T on either side, we eas-ily obtain ˆ y = y − R T n ( y T R T n + t T n + d ) . (7)By comparing to (3), it is obvious that ˆ n = R T n and ˆ d = t T n + d must represent the symmetry plane parameters forthe original, untransformed set Y .
4. Alignment and Symmetry as a Joint Opti-mization Problem
We now introduce our novel optimization objectivewhich jointly optimizes an aligning point set transforma-tion as well as the plane of symmetry. The objective is then3olved in a branch-and-bound optimization paradigm, forwhich we introduce both the domain parameterization aswell as the derivation of upper and lower bounds.
We still assume that our two partial object point sets aregiven by X = { x i } Mi =1 and Y = { y i } Ni =1 , and that thesymmetry plane is represented by the normal n and depth d . We define X s = { x si | x si = x i − n ( x Ti n + d ) , i =1 , · · · , M } to be the corresponding reflected point set of X .We furthermore define X r = { x ri | x ri = R T ( x i − t ) , i =1 , · · · , M } and Y r = { y ri | y ri = Ry i + t , i = 1 , · · · , N } tobe the aligned sets in either direction. The symmetry fittingobjective of X employs e si ( n , d | x i ) = (cid:107) x i − n ( x Ti n + d ) − x j (cid:107) , (8)where the difference to (4) is given by the fact that x j isnow the nearest neighbour from the set X (cid:83) Y r . Similarly,using equation (7), the symmetry objective function for Y employs e si (ˆ n , ˆ d | y i ) = (cid:107) y i − n ( y Ti ˆ n + ˆ d ) − y j (cid:107) , (9)where y j is now chosen as the nearest neighbour from theset X r (cid:83) Y . The final registration error itself employs e ri ( R , t | y i ) = w i (cid:107) Ry i + t − x j (cid:107) , (10)where x j is chosen as the nearest neighbour from the set X (cid:83) X s and weight w i is used to take the set X as alignedpoints first. The overall objective function becomes E ( R , t , n , d ) = M (cid:88) i =1 e si ( n , d | x i ) (11) + N (cid:88) i =1 (cid:110) e ri ( R , t | y i ) + e si (ˆ n , ˆ d | y i ) (cid:111) . Direct optimization over R , t , n , and d using traditionalICP would easily get trapped in the nearest local mini-mum. We therefore propose to minimize the energy objec-tive using the globally optimal branch-and-bound paradigm,an exhaustive search strategy that branches over the entireparameter space. In order to speed up the execution, themethod derives upper and lower bounds for the minimal en-ergy on each branch (i.e. sub-volume of the optimizationspace), and discards branches for which the lower boundremains higher than the upper bound in another branch. Inthe remainder of this section, we will discuss the two im-portant questions of (i) how to parametrize and branch theparameter space, and (ii) how to find concrete values for theupper and lower bounds.
2D problem : For a 2D point set, the domain of all feasi-ble alignment parameters is represented by an angle r to pa-rameterize the planar rotation R r = (cid:20) cos( r ) − sin( r )sin( r ) cos( r ) (cid:21) ,and a 2D translation vector t . The space of all 2D rota-tions can therefore be compactly represented by the interval [ − π, + π ] . For the translation, we assume that the optimaltranslation must lie within a cube defined by the interval [ − (cid:15), + (cid:15) ] . The symmetry plane in the 2D case becomes aline which is parameterized by an angle α defining the nor-mal vector n = [cos( α ) sin( α )] T , and a scale d . Given thedual representation of line normal vectors, α is constrainedto the interval [ − π , + π ] , and d lies in the interval [ − ε, + ε ] .
3D problem : The disadvantage of BnB is that its com-plexity grows exponentially in the dimensionality of theproblem. We therefore take prior information about the 3Dpoint sets into account that helps to decrease the dimen-sionality. We make the assumption that most objects arestanding upright to the ground plane. The rotation betweenthe different partial point sets is therefore still constrainedto be a 1D rotation about the vertical axis, and the normalvector n remains a 1D variable the horizontal plane. Thetranslation however becomes a 3D vector over the interval [ − (cid:15), + (cid:15) ] .In conclusion, the 2D alignment problem has 5 de-grees of freedom, whereas the 3D problem turns into a -dimensional estimation problem. The basic idea of BnB is to partition feasible set intoconvex sets which means it crucially depends on upper andlower bounds for the objective L E and the lower bound E of the optimal, joint L registration and symmetry cost E ∗ on a given interval of variables are therefore given as E . = M (cid:88) i =1 e si ( α, d | x i ) + N (cid:88) i =1 { e ri ( r, t | y i ) + e si ( r, t , α, d | y i ) } (12) E . = M (cid:88) i =1 e si ( α, d | x i ) + N (cid:88) i =1 { e ri ( r, t | y i ) + e si ( r, t , α, d | y i ) } (13) Upper bounds on an interval are easily given by the energyof an arbitrary point within the interval. Given an interval4entered at { r , t , α , d } , upper bounds are therefore eas-ily defined as e si ( α, d | x i ) = e si ( α , d | x i ) e ri ( r, t | y i ) = e ri ( r , t | y i ) e si ( r, t , α, d | y i ) = e si ( r , t , α , d | y i ) . (14) The remainder of this section discusses the derivation of thelower bounds.
Lower bound for the alignment error e ri ( r, t | y i ) : Fora rotation interval of half-length σ r with centre r , we have (cid:107) R r y − R r y (cid:107) ≤ sin ( min ( σ r / , π/ (cid:107) y (cid:107) . = γ r (cid:107) y (cid:107) . (15) γ r is also called the rotation uncertainty radius. Proof : Using Lemmas 3.1 and 3.2 of [13], we have: (cid:107) R r y − R r y (cid:107) = 2 sin ( ∠ ( R r y , R r y ) / (cid:107) y (cid:107)≤ sin ( min ( ∠ ( R r , R r ) , π ) / (cid:107) y (cid:107)≤ sin ( min ( | r − r | , π ) / (cid:107) y (cid:107)≤ sin ( min ( σ r / , π/ (cid:107) y (cid:107) . (16) We can similarly derive a translation uncertainty radius γ t . For a translation volume with half side-length σ t cen-tered at t , we have (cid:107) ( x + t ) − ( x + t ) (cid:107) = (cid:107) t − t (cid:107) ≤ √ σ t . = γ t (17) Note that in the 2D case, we have γ t . = √ σ t . The lowerbound of the registration term in equation (10) becomes e ri ( r, t | y i ) ≥ e ri ( r , t | y i ) − w i ( γ r (cid:107) y i (cid:107) + γ t ) . = e ri ( r, t | y i ) (18) For more details, we kindly refer the reader to [13, 26].
Lower bound of symmetry term e si ( α, d | x i ) : Assum-ing that the normal is defined by an α -interval of half-length σ α and with center α , we have (cid:107) x T ( n − n ) (cid:107) ≤ (cid:107) n − n (cid:107)(cid:107) x (cid:107) = (cid:113) (1 − cos ( α − α )) + sin ( α − α ) (cid:107) x (cid:107)≤ (cid:112) − cos( σ α )) (cid:107) x (cid:107) . = γ α (cid:107) x (cid:107) . (19) For the depth d ∈ [ d − σ d , d + σ d ] , we simply have | d − d | ≤ σ d . = γ d . (20) Now let x j ∈ X ∪Y r be the closest point to ( x i − n ( x Ti n + d )) , and let x j ∈ X ∪ Y r be the closest point to ( x i − n ( x Ti n + d )) . The lower bound is derived as follows: e si ( α, d | x i ) = (cid:107) x i − n ( x Ti n + d ) − x j (cid:107) = (cid:107) x i − n ( x Ti n + d ) − x j − (cid:16) n ( x Ti n + d ) − n ( x Ti n + d ) (cid:17) (cid:107)≥ (cid:107) x i − n ( x Ti n + d ) − x j (cid:107)− (cid:107) n ( x Ti n + d ) − n ( x Ti n + d ) (cid:107)≥ e si ( α , d | x i ) − (cid:107) x Ti nn − x Ti n n (cid:107) − (cid:107) n d − n d (cid:107) (21) We furthermore have (cid:107) x Ti nn − x Ti n n (cid:107) = (cid:107) x Ti nn − x Ti n n + x Ti n n − x Ti n n (cid:107)≤ | x Ti n − x Ti n | · (cid:107) n (cid:107) + | x Ti n | · (cid:107) n − n (cid:107)≤ (cid:107) x i (cid:107) · (cid:107) n − n (cid:107) = 2 γ α (cid:107) x i (cid:107) (22) and (cid:107) n d − n d (cid:107) = (cid:107) n d − n d + n d − n d (cid:107)≤ | d − d | + | d |(cid:107) n − n (cid:107) ≤ γ d + | d | γ α . (23) Substituting (22) and (23) in (21), we finally obtain e si ( α, d | x i ) . = max ( e si ( α , d | x i ) − γ α (cid:107) x i (cid:107) + γ d + | d | γ α ) , . (24) Lower Bound of symmetry term e si ( r, t , α, d | y i ) : Byusing ˆ n = R T n and ˆ d = t T n + d , we analogously derive e si ( r, t , α, d | y i ) ≥ e si ( r , t , α , d | y i ) − (cid:107) y Ti ˆ n ˆ n − y Ti ˆ n ˆ n (cid:107) − (cid:107) ˆ n ˆ d − ˆ n ˆ d (cid:107) (25) By substituting ˆ n = R T n , similar to 22, the first term gives (cid:107) y Ti R T nR T n − y Ti R T n R T n (cid:107)≤ (cid:107) y i (cid:107)(cid:107) R T n − R T n (cid:107) = 2 (cid:107) y i (cid:107)(cid:107) R T n − R T n + R T n − R T n (cid:107)≤ (cid:107) y i (cid:107) ( γ α + γ r ) . (26) By also substituting ˆ d = t T n + d , the second term gives (cid:107) ( t T n + d ) R T n − ( t T n + d ) R T n (cid:107)≤ | t T n + d | · (cid:107) R T n − R T n (cid:107) + (cid:107) R T n (cid:107)(cid:107) t T n − t T n (cid:107) + (cid:107) R T n (cid:107)| d − d |≤ ( γ α + γ r ) | t T n + d | + (cid:107) t (cid:107) γ α + γ t + γ d , (27) where we have used (cid:107) t T n − t T n (cid:107) = (cid:107) t T n − t T n + t T n − t T n (cid:107) ≤ (cid:107) t (cid:107)(cid:107) n − n (cid:107) + (cid:107) n (cid:107)(cid:107) t − t (cid:107) . Substituting(26) and (27) in (25), we finally obtain e si ( r, t , α, d | y i ) . = max ( e si ( r , t , α , d | y i ) − γ t + γ d + ( γ α + γ r ) (cid:16) (cid:107) y i (cid:107) + | t T n + d | (cid:17) + (cid:107) t (cid:107) γ α ) , . (28)
5. Implementation
Similar to prior art [26], we improve the algorithm’s abil-ity to handle the dimensionality of the problem by installinga nested BnB paradigm.
We install a nested BnB scheme in which the outer layersearches through the space C rα of all angular parameters,while the inner layer optimizes over the space C t d of trans-lation and depth. While finding the bounds in a sub-volume5 lgorithm 1 Main Algorithm: BnB search for optimal reg-istration and symmetry parameters
Input:
Data and model point set; threshold τ ; initial inter-vals C rα and C t d . Output:
Globally minimal error E ∗ and the optimal r ∗ , t ∗ , α ∗ , d ∗ Put C rα into priority queue Q rα . Set E ∗ = + ∞ . loop Read out interval with lowest E rα from Q rα .Quit the loop if E ∗ − E rα < τ .Divide interval into 4 sub-intervals. for each sub-interval C rα do Compute the corresponding optimal t and d by call-ing Algorithm 2 with R r and n (zero rotation andnormal uncertainty).Compute E rα and E rα for C rα with the optimal t , d . if E rα < E ∗ then Run ICP with initialization of ( r , t , α , d ) Update E ∗ , r ∗ , α ∗ , t ∗ , and d ∗ with the results ofICP. end if Discard C rα if E rα ≥ E ∗ ; otherwise put it into Q rα end forend loop of the angle space, the algorithm calls the inner BnB al-gorithm to identify the optimal translation and scale. Oneimportant approximation that accelerates the execution isthat—whenever the bounds in a sub-volume are derived—the uncertainty of the non-optimized parameters of that par-ticular layer are set to zero. Detailed descriptions are givenin Algorithm 1 (the Main Algorithm) and Algorithm 2 (theInner BnB). Within the outer layer, whenever BnB identifies an inter-val C rα with an improved upper bound, we will execute aconventional local ICP algorithm starting from the center ofthe C rα and taking t ∗ and d ∗ as an initial value. Once ICPconverges to a local minimum with a lower function value,the new value is used to further reduce the upper bound.The technique is inspired by Yang et al. [27]. A general problem with partially overlapping point setsis that—even at the global optimum—some points may sim-ply not have a correspondence, and should hence be treatedas outliers. Although the addition of symmetry and point re-flections already greatly alleviates this problem, we still addthe strategy proposed in Trimmed ICP [7] for robust point-set registration. More specifically, in each iteration, only
Algorithm 2
BnB search for optimal translation and depthgiven rotation and normal
Input:
Data and model point set; threshold τ ; initial inter-vals C t d ; Currently lowest error E ∗ Output:
Minimal error E ∗ and the optimal t ∗ , d ∗ Put C t d into priority queue Q t d . loop Read out interval with lowest E t d from Q t d .Quit the loop if E ∗ − E t d < τ .Divide interval into 8(2D)/16(3D) sub-intervals. for each sub-interval C t d do Compute E t d and E t d for C t d with the r , n . if E t d < E ∗ then Update E ∗ and t ∗ , d ∗ . end if Discard C t d if E t d ≥ E ∗ ; otherwise put it into Q t d end forend loop a subset of the matched data points with smallest point-to-point distances are used for motion computation. In thiswork, we choose a -subset for both symmetry and reg-istration residuals.
6. Experiments
We now report our experimental results on both syntheticand real data. In all our experiments, we pre-normalizedthe pointsets such that all points are within the domain of [ − , . The parameter ε is set to . (cf. Section 4.2).We run experiments on both 2D and 3D data, and com-pare our results against the open-source implementation ofGo-ICP complemented by the ransac-based symmetry de-tection method presented in [10] and applied to a fusion ofthe aligned point sets. For Go-ICP, the stopping criterion τ is set to . · . · N , and for our method it is set to . · . · ( M + 2 N ) where . is the trimming ratio, M, N are the number of points in X and Y , respectively.For [10], the number of iterations is limited to . Each experiment is generated by taking an image thatcontains a symmetrical object, and using the Sobel edge de-tector to extract the object contour points. To evaluate theperformance, we randomly divide the contour points intotwo subsets with a defined and structured overlap. To con-clude, Y is transformed by a random rotation and translationdrawn from the intervals ± degrees and ± . . Handling of limited overlap : The overlap between bothpoint sets if varied from to . For each overlap ra-tio, we repeat 50 experiments each time choosing a randomobject, rotation and translation. Fig 1 shows an exampleresult, where the left is the result of our proposed method,6 ur result symmetry axis
2D Go-ICP symmetry axis ground truth symmetry axis
Figure 1. Example of a 2D point set registration with an overlapratio of . . (a). overlap ratio a v e r age deg r ee e rr o r mean rotation error of our methodmedian rotation error of our methodmean rotation error of 2D Go-ICPmedian rotation error of 2D Go-ICPmean normal error of our methodmedian normal error of our methodmean normal error of ransac-based methodmedian normal error of ransac-based method (b). overlap ratio a v e r age L - e rr o r mean translation error of our methodmedian translation error of our methodmean translation error of 2D Go-ICPmedian translation error of 2D Go-ICPmean depth error of our methodmedian depth error of our methodmean depth error of ransac-based methodmedian depth error of ransac-based method Figure 2. Mean and median errors of 2D registration comparedagainst 2D Go-ICP followed by ransac-based symmetry detec-tion [10]. (a) shows angular errors for the rotation and the sym-metry plane normal, while (b) shows the errors of the translationand the depth of plane. the centre is the result of the 2D version of Go-ICP, and theright one shows the ground truth alignment. Figure 2 showsall mean and median errors of all optimisation parametersover all evaluation results. As can be observed, our methodhas a substantially better ability to deal with partial over-laps compared to Go-ICP, thus leading to vastly improvedsymmetry plane parameters as well.
Outlier handling : To test resilience against outliers, we re-peat the same experiment but add up to outliers to both X and Y . Figure 3 indicates an example result, and Figure 4again illustrates the mean and median errors over all exper-iments. While the registration error starts to increase earlierand the average angular errors tend to be higher, it can stillbe concluded that our method significantly outperforms Go-ICP followed by symmetry plane detection using [10]. We choose symmetrical CAD models fromShapeNet [1], different types from classes. There objects contain more than one symmetry plane. For eachobject, we generate depth images (with occlusions) fromrandom views around the object, and producing pairsof point sets for each object instance with varying overlap,and finally a total of about point-set registration our result symmetry axis
2D Go-ICP symmetry axis ground truth symmetry axis
Figure 3. Example of 2D point set registration with an overlap ratioof . and outliers. (a) overlap ratio a v e r age deg r ee e rr o r mean rotation error of our methodmedian rotation error of our methodmean rotation error of 2D Go-ICPmedian rotation error of 2D Go-ICPmean normal error of our methodmedian normal error of our methodmean normal error of ransac-based methodmedian normal error of ransac-based method (b) overlap ratio a v e r age L - e rr o r mean translation error of our methodmedian translation error of our methodmean translation error of 2D Go-ICPmedian translation error of 2D Go-ICPmean depth error of our methodmedian depth error of our methodmean depth error of ransac-based methodmedian depth error of ransac-based method Figure 4. Mean and median errors of 2D registration using thesame methods and up to outliers. (a) shows angular errorsfor the rotation and the symmetry plane normal, while (b) showsthe errors of the translation and the depth of plane. experiments. Our result is shown in Figure 5, whichagain illustrates mean and median errors of all estimatedquantities as a function of the overlap between the sets.In 3D, it is natural that the camera captures only a smallpart of the object, and that in turn even the fusion of bothaligned point scans may not enable a stable estimation ofthe symmetry plane, hence the slightly increased meanerror. However, especially the median error is still lowercompared to Go-ICP, thus confirming the mutual benefitsof our joint estimation paradigm. Figure 6 shows a fewconcrete examples for which the 3D objects only contain asingle symmetry plane and for which our method is largelyoutperformed. Figure 7 shows failure examples where thepartial point sets lead to an ambiguity in the symmetryplanes. In particular, in the first example, Go-ICP stillworks as the overlap ratio is sufficient for the registration.
Our last experiment is an exciting application to real datathat goes back to the initial motivation in the introduction.Figure 8 shows two different depth images captured by aKinect camera, each one containing three instances of thesame object under different orientations. By pairwise align-ment of partial object scans, the mutual information is trans-ferred thus leading to more complete perception of each in-dividual object. Note that we use simple ground plane fit-7 .1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 (a). overlap ratio a v e r age deg r ee e rr o r mean rotation error of our methodmedian rotation error of our methodmean rotation error of 3D Go-ICPmedian rotation error of 3D Go-ICPmean normal error of our methodmedian normal error of our methodmean normal error of ransac-based methodmedian normal error of ransac-based method (b). overlap ratio a v e r age L - e rr o r mean translation error of our methodmedian translation error of our methodmean translation error of 3D Go-ICPmedian translation error of 3D Go-ICPmean depth error of our methodmedian depth error of our methodmean depth error of ransac-based methodmedian depth error of ransac-based method Figure 5. Mean and median errors of 3D registration comparedagainst 3D Go-ICP followed by ransac-based symmetry detec-tion [10]. Note that no outliers are added, but–as for the case ofthe results in 2D–a similar trend has been confirmed for up to 30%outliers. ting and depth discontinuity-aware point clustering withinobject bounding boxes to isolate the partial object scans.With known position of the ground plane, we then trans-form the whole scene to be orthogonal to the ground planeand meet the assumption that all objects are placed uprightand—in terms of relative rotation—differ only by an angleabout the vertical axis. In Figure 8, the first column showsthe original scan in different orientations, the second onethe partial object measurements, the third one the comple-tion obtained by using Go-ICP as an alignment algorithmand ransac-based symmetry detection, and the last one theresult obtained by using our algorithm. As can be observed,our joint alignment strategy outperforms the comparisonmethod, and achieves meaningful shape completion.
7. Discussion
Symmetry detection and point set alignment over setswith small overlap are challenging problems if handled sep-arately. Our work demonstrates a substantial improvementin both accuracy and success rate of the alignment by solv-ing those two problems jointly. The information gainedfrom estimating symmetry and reflecting points notablymakes up for otherwise missing correspondences. How-ever, our current implementation is not competitive in termsof running time, hence we are working on a parallel imple-mentation. We furthermore plan to extend the algorithm tomulti-point set registration, and improve its ability to dealwith the situation of multiple symmetry planes.
Figure 6. 3D alignment results for concrete experiments. Overlapratio and added outliers are each pair indicated.Figure 7. Example of failure cases where there is ambiguity in thesymmetry plane.
References [1] L. Guibas P. Hanrahan Q. Huang Z. Li S. Savarese M. SavvaS. Song H. Su et al. A. X. Chang, T. Funkhouser. Shapenet:An information-rich 3d model repository. arXiv preprintarXiv:1512.03012 , 2015. 7[2] K.S. Arun, T.S. Huang, and S.D. Blostein. Least-Squares Fit-ting of Two 3-D Point Sets.
Transations of Pattern Analysisand Machine Intelligence , 9(5):698–700, 1987. 2[3] Paul J Besl and Neil D McKay. Method for registration of3-d shapes. In
Sensor fusion IV: control paradigms and datastructures , volume 1611, pages 586–606. International Soci-ety for Optics and Photonics, 1992. 2[4] Dylan Campbell and Lars Petersson. Gogma: Globally-optimal gaussian mixture alignment. In
Proceedings of theIEEE Conference on Computer Vision and Pattern Recogni-tion , pages 5685–5694, 2016. 2 igure 8. Shape completion by alignment and transfer of seg-mented, partial object scans in RGBD images. The center columnshows the completion results obtained by applying Go-ICP, whilethe right column shows the result obtained by our method.[5] Jia Chen, Xiaojun Wu, Michael Yu Wang, and Xuanfu Li.3d shape modeling using a self-developed hand-held 3d laserscanner and an efficient ht-icp point cloud registration algo-rithm. Optics & Laser Technology , 45:414–423, 2013. 1[6] Yang Chen and G´erard Medioni. Object modelling by regis-tration of multiple range images.
Image and vision comput-ing , 10(3):145–155, 1992. 2[7] Dmitry Chetverikov, Dmitry Stepanov, and Pavel Krsek. Ro-bust euclidean alignment of 3d point sets: the trimmed iter-ative closest point algorithm.
Image and vision computing ,23(3):299–309, 2005. 2, 6[8] Haili Chui and Anand Rangarajan. A new point matchingalgorithm for non-rigid registration.
Computer Vision andImage Understanding , 89(2-3):114–141, 2003. 2[9] Marcelo Cicconet, David GC Hildebrand, and Hunter El-liott. Finding mirror symmetry via registration and optimalsymmetric pairwise assignment of curves: Algorithm and re-sults. In
Proceedings of the IEEE International Conferenceon Computer Vision , pages 1759–1763, 2017. 3[10] Andrea Cohen, Christopher Zach, Sudipta N Sinha, andMarc Pollefeys. Discovering and exploiting 3d symme-tries in structure from motion. In , pages 1514–1521. IEEE, 2012. 2, 6, 7, 8[11] Martin A Fischler and Robert C Bolles. Random sampleconsensus: a paradigm for model fitting with applications toimage analysis and automated cartography.
Communicationsof the ACM , 24(6):381–395, 1981. 2[12] S´ebastien Granger and Xavier Pennec. Multi-scale em-icp:A fast and robust approach for surface registration. In
Eu-ropean Conference on Computer Vision , pages 418–432.Springer, 2002. 2[13] Richard I Hartley and Fredrik Kahl. Global optimizationthrough rotation space search.
International Journal of Com-puter Vision , 82(1):64–79, 2009. 5[14] M. Bloesch A. Davison J. McCormac, R. Clark and S.Leutenegger. Fusion++: Volumetric object-level slam. In In IEEE International Conference on 3D Vision (3DV) , year= 2018 . 1[15] Bing Jian and Baba C Vemuri. Robust point set registra-tion using gaussian mixture models.
IEEE transactions onpattern analysis and machine intelligence , 33(8):1633–1645,2010. 2[16] Wei Jiang, Kai Xu, Zhi-Quan Cheng, and Hao Zhang.Skeleton-based intrinsic symmetry detection on point clouds.
Graphical Models , 75(4):177–188, 2013. 3[17] Hongdong Li and Richard Hartley. The 3d-3d registrationproblem revisited. In , pages 1–8. IEEE, 2007. 2[18] Yanxi Liu, Hagit Hel-Or, Craig S Kaplan, Luc Van Gool,et al. Computational symmetry in computer vision and com-puter graphics.
Foundations and Trends R (cid:13) in ComputerGraphics and Vision , 5(1–2):1–195, 2010. 2[19] Andriy Myronenko and Xubo Song. Point set registration:Coherent point drift. IEEE transactions on pattern analysisand machine intelligence , 32(12):2262–2275, 2010. 2[20] R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D.Kim, A. J. Davison, P. Kohli, J. Shotton, S. Hodges, andA. Fitzgibbon. KinectFusion: Real-time dense surface map-ping and tracking. In
International Symposium on Mixed andAugmented Reality , 2011. 1[21] Carl Olsson, Fredrik Kahl, and Magnus Oskarsson. Branch-and-bound methods for euclidean registration problems.
IEEE Transactions on Pattern Analysis and Machine Intel-ligence , 31(5):783–794, 2008. 2[22] Alvaro Parra Bustos, Tat-Jun Chin, and David Suter. Fast ro-tation search with stereographic projections for 3d registra-tion. In
Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition , pages 3930–3937, 2014. 2[23] Kari Pulli. Multiview registration for large data sets. In
Sec-ond International Conference on 3-D Digital Imaging andModeling (Cat. No. PR00062) , pages 160–168. IEEE, 1999.1[24] Yanghai Tsin and Takeo Kanade. A correlation-based ap-proach to robust point set registration. In
European confer-ence on computer vision , pages 558–569. Springer, 2004. 2[25] Kai Xu, Hao Zhang, Andrea Tagliasacchi, Ligang Liu, GuoLi, Min Meng, and Yueshan Xiong. Partial intrinsic reflec-tional symmetry of 3d shapes.
ACM Transactions on Graph-ics (TOG) , 28(5):138, 2009. 3[26] Jiaolong Yang, Hongdong Li, Dylan Campbell, and YundeJia. Go-icp: A globally optimal solution to 3d icp point-set registration.
IEEE transactions on pattern analysis andmachine intelligence , 38(11):2241–2254, 2015. 1, 2, 5[27] Jiaolong Yang, Hongdong Li, and Yunde Jia. Go-icp: Solv-ing 3d registration efficiently and globally optimally. In
Pro-ceedings of the IEEE International Conference on ComputerVision , pages 1457–1464, 2013. 2, 6[28] Zhengyou Zhang. Iterative point matching for registrationof free-form curves and surfaces.
International journal ofcomputer vision , 13(2):119–152, 1994. 1, 2, 3, 13(2):119–152, 1994. 1, 2, 3