Complete Endomorphisms in Computer Vision
CComplete Endomorphisms in Computer Vision
Applied to multiple view geometry
Finat, Javier · Delgado-del-Hoyo, Francisco
February 24, 2020
Abstract
Correspondences between k -tuples of pointsare key in multiple view geometry and motion analy-sis. Regular transformations are posed by homographiesbetween two projective planes that serves as structuralmodels for images. Such transformations can not in-clude degenerate situations. Fundamental or essentialmatrices expand homographies with structural infor-mation by using degenerate bilinear maps. The projec-tivization of the endomorphisms of a three-dimensionalvector space includes all of them. Hence, they are ableto explain a wider range of eventually degenerate trans-formations between arbitrary pairs of views. To includethese degenerate situations, this paper introduces a com-pletion of bilinear maps between spaces given by anequivariant compactification of regular transformations.This completion is extensible to the varieties of fun-damental and essential matrices, where most methodsbased on regular transformations fail. The constructionof complete endomorphisms manages degenerate pro-jection maps using a simultaneous action on source andtarget spaces. In such way, this mathematical construc-tion provides a robust framework to relate correspond-ing views in multiple view geometry. Keywords
Epipolar Geometry · Essential Matrix · Fundamental Matrix · Degeneracies · Secant Varieties · Adjoint Representation
MoBiVAP research groupUniversidad de ValladolidCampus Miguel DelibesPaseo de Bel´en, 1147010, Valladolid, SpainTel.: +34-983-184-398
E-mail: franciscojavier.fi[email protected]
A classical issue regarding 3D reconstruction and mo-tion analysis concerns the preservation of the continuityof the scene or the flow, although small changes in inputoccurs. There are a lot of answers where different con-straints have been introduced from the early eighties.The most common constraint is the structural bilinear- or multilinear - tensor created with k -tuples of corre-sponding elements (points and/or lines) for the camerapose. All these constraints are very sensitive to noise.Classical approaches are based on minimal solutions ex-tracted from noise measurements following a RANSAC scheme. However, indeterminacies persist for some de-generate situations - created by low rank matrices - thatcan arise for mobile cameras.Classic literature in computer vision [4,9] displaya low attention to degenerate cases in structural mod-els. These cases arise when independence conditions forfeatures are not fulfilled, including situations where thecamera turns around its optical axis or it is in fronto-parallel position w.r.t. a planar surface (a wall or theground, e.g.). Then, the problem is ill-posed, and con-ventional solutions consists in performing a “small per-turbation” or reboot the process. Both strategies dis-play issues concerning the lack of control about theperturbation to be made that generate undesirable dis-continuities. Thus, it is important to develop alterna-tive strategies which can maintain some kind of “coher-ence” by reusing the “recent history” of the trajectory.History is continuously modeled in terms of genericallyregular conditions for tensors in previously sampled im-ages with a discrete approach of a well-defined path inthe space of structural tensors. Unfortunately, degener- Random Sample Consensus a r X i v : . [ c s . C V ] F e b Finat, Javier, Delgado-del-Hoyo, Francisco acy conditions for typical features give indeterminacyfor limits of structural tensors, which must be removed.Our approach consists of considering Kinematic infor-mation of the matrix version of the gradient field forindeterminacy loci.Less attention has been paid to preserve the “conti-nuity” of eventually singular trajectories in the space ofbilinear maps linked to the automatic correspondencebetween pairs of views. In this case, singular maps arethe responsible for indeterminacies in tensors and lie onsingular strata of the space of bilinear maps[24]. In thiswork, we develop a more down-to-earth approach usingsome basic properties of the projectivization of spacesof endomorphisms, including homographies H , funda-mental F and essential E matrices. All of them can bedescribed in terms of orbits by a group action on thespace of endomorphisms End ( V ), i.e. linear maps of avector space V in itself. Their simultaneous algebraictreatment allows to extend the algebraic completion tothe space of eventually degenerated central projectionmaps M C with center C . Intuitively, the key for thecontrol of degenerate cases is to select appropriate lim-its of tangents in a “more complete” space.Therefore, the main goal of our work is to modela continuous solution that also considers degeneratedcases for simplest tensors (fundamental and essentialmatrices, e.g.) such those appearing in structural mod-els for 3D Reconstruction. In order to achieve this goal,we introduced an “equivariant compactification” of thespace of matrices w.r.t. group actions linked to kine-matic properties visualized in the dual space. This dou-ble representation (positions and “speed”) stores the“recent history” represented by a path in the tangentbundle τ End ( V ) to End ( V ) for the trajectory of a mobilecamera C ( t ). Our approach is based on a dual presen-tation for the rank stratification of matrices. This dualrepresentation encodes tangential information at eachpoint represented by the adjoint matrix, a 3 × ∇ ( det ( A )) of thedeterminant of A .In the simplest case, after fixing a basis B V of V , en-domorphisms of a 3D vector space V (cid:39) R are given byarbitrary 3 × r ≤
3. In partic-ular, from the differential viewpoint, sets of homogra-phies H and regular (i.e. rank 2) fundamental matrices F can be considered as two G -orbits of the Lie algebra g = End ( V ) of G = GL (3) = Aut ( V ) by the action ofthe projectivization P GL (3) of the general linear groupcorresponding to rank 3 and rank 2 matrices, respec-tively. More generally, the description of End ( V n +1 ) asa union of orbits by the action of GL ( n + 1; R ) givesa structure as an “orbifold”, i.e. a union of G -orbits containing their degenerate cases, which are usuallyexcluded from the analysis. They are recovered by in-troducing a “compactification” where degenerate casesare managed in terms of successive envelopes by linearsubspaces W of V . All arguments can be extended tohigher dimensions and even to hypermatrices represent-ing more sophisticated tensors. However, for simplifica-tion purposes, we constraint ourselves only to endomor-phisms extending planar or spatial homographies to thesingular case.The rest of the paper is organized as follows. Section2 provides the mathematical background to understandthe rest of the paper and frames our approach in thestate of the art. Section 3 analyzes the simplest casesincluding regular transformations defined by homogra-phies for the planar case that relate 2D views using thefundamental variety. Section 4 extends the approachto rigid transformations in the third dimension, includ-ing metric aspects in terms of the essential variety in-volving source and target spaces in P . Section 5 stud-ies the structural connection between them using thesimultaneous action on source and target spaces of avariable projection linked to the camera pose; left-right A and contact K -equivalences are explained. Section 6provides additional insight concerning the details andpractical considerations for implementing this approachin VO systems. Finally, Section 7 concludes the paperwith a summary of the main results and guidelines forfurther research. Local symmetries are ubiquitous in a lot of problemsin Physics and Engineering involving propagation phe-nomena. Most approaches in applied areas consider onlyregular regions, by ignoring any kind of degenerationslinked to rank deficient matrices linked to linearizationof phenomena. To include them, we develop a “locallysymmetric completion” of eventually singular transfor-mations for involved tensors such those appearing inReconstruction issues. Our approach is not quite origi-nal; a similar idea can be found in [26], which introducesa locally symmetric structure in a differential frame-work concerning geodesics on the essential manifold.Nevertheless, the initial geometric description as sym-metric space (union of orbits linked to the rank preser-vation) can not be extended to include a differentialapproach to degenerate cases. Due to the occurrence ofsingularities, the support given as a quotient variety isnot a smooth manifold but a singular algebraic variety Visual Odometryomplete Endomorphisms in Computer Vision 3 where the methods for Riemannian manifolds no longerapply.A larger description of a locally symmetric struc-ture may be performed by extending the ordinary al-gebraic approach. Roughly speaking, it suffices to addlimits of tangent subspaces along different “branches”at singularities and “extend the action” to obtain amore complete description including kinematic aspects.In this way, one obtains a “local replication” compati-ble with the presence of singularities in the adherence of“augmented” orbits by tangent spaces at regular points.So, for the “subregular” case - codimension one orbit- it suffices to construct pairs of eventually degeneratetransformations involving the original one and a “gen-eralized dual” transformation (given by the adjoint mapin the regular case) representing neighbor tangent di-rections. So, first order differential approaches of even-tually degenerate maps allow to propagate - and conse-quently, anticipate - partial representations of expectedviews, even in presence of rank-deficient matrices.This extended duality allows a simultaneous treat-ment of incidence and tangency conditions (both areprojectively invariant), and to manage degenerate casesin terms of “complete objects” as limits in an enlargedspace (including the original space and their duals) whichcan be managed as a locally symmetric space in termsof extended transformations (original ones and their ex-terior powers). Besides its differential description as agradient in the space of matrices, a more geometric de-scription can be developed in terms of pairs of loci andtheir envelopes. The extended transformations act onthe source or ambient space (right action), and on itsdual space which can be considered as a target spacerepresenting envelopes by tangent subspaces. This ideais reminiscent of the contact action K which preservesthe graph and it provides a natural extension of theright-left action A (see next paragraph). A discrete ver-sion of last action has been used by Kanade, Tomasiand Lucas KLT along the early nineties in regard toStructure from Motion approaches to 3D Reconstruc-tion. Both actions are commonly used for the infinites-imal classification of map-germs in differential classifi-cation of map-germs. However, our approach is morefocused towards a local description of the space of gen-eralized transformations and/or projection maps as alocally symmetric space. This structure has the addi-tional advantage of allowing the extension of Rieman-nian properties given in terms of geodesics.The simplest simultaneous action on source and tar-get spaces is the Cartesian or direct product of ac-tions. It is denoted by the A -action where A := R ×L is the right-left action. Its orbits are given by the Kanade-Lucas-Tomasi double conjugacy classes from the algebraic viewpoint.The A -action is very useful for decoupled models (im-plicit in KLT algorithms or SfM , e.g.), and conse-quently very useful by computational reasons. Despitethe wide interest for the above approaches, the A -actionis less plausible than the K -action, which incorporatesthe graph preservation (corresponding to quadratic con-tact between a manifold M and its tangent space T p M at each contact point p ∈ M ) as the structural con-straint.In our case, contact equivalence is based on a cou-pling between images and scene representations. Al-though contact equivalence is well known in Local Dif-ferential Topology, its use in Computer Vision is veryscarce. It is implicitly embedded in some recognition ap-proaches where one exchanges information about con-trol points and envelopes. However, to our best knowl-edge, it has not been applied to multiple view geometryissues. We constraint ourselves to almost generic phe-nomena given by low-corank c ≤ This section extends conventional homographies to in-clude the degenerate cases by considering arbitrary -including eventually singular - endomorphisms (i.e. lin-ear maps on a vector space) acting on configurationsof points. In particular, regular transformations up toscale of a 3-dimensional vector space V belong to thegroup of homographies which is an open subset of theprojective space P = P End ( V ). Its complementaryis the set of singular endomorphisms up to scale, acubic hypersurface defined by det ( X ) = 0 for X =( x ij ) ≤ i,j ≤ and containing the fundamental subvariety F and the essential manifold E .Planar homographies represent regular transforma-tions between two projective planes P = P V of 2Dviews. Thus, any homography is an element of the pro-jectivized linear group P GL (3 , R ), where GL (3 , R ) isthe general linear group acting on the projective modelof each view. Given a reference for V , each elementof GL (3 , R ) can be represented by a (3 ×
3) regularmatrix, i.e. with non-vanishing determinant. By con-struction, homographies (regular transformations up toscale) can not include degenerate transformations suchthose appearing in fundamental or essential matrices.Then, these matrices can be considered as “degenerate”endomorphisms (represented by defficient rank matri- Structure from Motion Finat, Javier, Delgado-del-Hoyo, Francisco ces up to scale) of an abstract real 3D space V with P V = P .Fundamental matrices F ∈ F are defined by degen-erate bilinear forms xFx (cid:48) = 0 linking pairs ( x , x (cid:48) ) ofcorresponding points. The set of pairs of correspond-ing points is called the join of two copies of P . Thisjoin is defined by the image of the Segre embedding s , : P × P (cid:44) → P giving a four-dimensional varietyof P that determines the 7D subvariety F of singularendomorphisms up to scale. Addition of the singularcases “completes” the homographies (regular transfor-mations), treating fundamental and essential matricesas degenerate transformation between two projectiveplanes inside the set of a completion or homogeneousendomorphisms.To understand how transformations can be extendedfrom a geometric to a kinematic framework, it is con-venient to introduce the differential approach for theregular subset. In terms of algebraic transformations,one mus replace the Lie group G of regular transforma-tions by its Lie algebra g := T e G where e is the neutralelement of F (the identity matrix for matrix groups); inparticular, T e Aut ( V ) = End ( V ). As usual in Lie the-ory, A , B , C . . . denote the elements of the group G , and X , Y , Z , . . . the elements of its Lie algebra g := T e G . Inparticular, any endomorphism X ∈ g (cid:96) (3) := T I GL (3)can be described by a matrix representing a point x ∈ P up to scale.The exponential map exp : g → G is a local diffeo-morphism (with the logarithm as inverse) that can beapplied to degenerate matrices for n = 3. In general,the set of homographies is an open set of P N where N = ( n + 1) −
1, whose complementary is given by thealgebraic variety of degenerate matrices, i.e. D := { D ∈ P N | det ( D ) = 0 } ), where det ( D ) is the determinantof D . We are interested in a better understanding of de-generacy arguments from the analysis of pencils (in facttangent directions) passing through a lower rank endo-morphism. The “moral” consists of the following simpleremark: the original action given by a matrix product,induces an action on linear ( k + 1) − dimensional sub-spaces by means the ( k + 1)-exterior power of the orig-inal action. Next paragraph illustrates this idea with asimple example.In particular, a line L represents a pencil (uni-parameterfamily) of endomorphisms { H λ } λ ∈ P , i.e. a linear tra-jectory in the ambient space P N where N = ( n +1) − n + 1 degenerate endomorphismscorresponding to the intersection L ∩ D denoted by D , D , . . . , D n +1 ∈ D . Inversely, the generic elementof the linear pencil µ i D i + µ j D j for 1 ≤ i < j ≤ n + 1 is a homography away from the variety D of degenerateendomorphisms .For n = 2, the intersection L ∩ D of a general pro-jective line L ⊂ P with the cubic algebraic variety D defined by det ( D ) = 0 gives generically three differ-ent degenerate endomorphisms. In particular, if L istangent to D at least two elements of L ∩ D can coa-lesce. An ordinary tangency condition is represented by2 d + d (cid:48) , where 2 d (resp. d (cid:48) ) represents a tangency (resp.simple) contact point corresponding to the intersectionof L with D . Linear pencils of matrices representingendomorphisms are interpreted as secant lines in theprojective ambient space.In general, k -secant varieties Sec ( k ; X ) to a vari-ety X ⊂ A n are defined by the set of points lying inthe closure of k -dimensional subspaces L k generated by( k + 1)-tuples of affine independent points generating k linearly independent vectors. They can be formallyconstructed by using the k -th exterior power ∧ k V ofthe underlying vector space V that allows to manage k + 1-tuples of points for k ≥
1. This statement canbe adapted to the underlying vector space of the Liealgebra g := T e G with its natural stratification by therank of any classical group G . The locally symmetricstructure is the key for extending the concept in thepresence of singularities. Although this construction isgeneral for End ( V ), it can be constrained to manageeventually degenerate tensors. Actually, this approachallows to connect old based-perspective methods usinghomographies with tensor-based methods.3.1 Fundamental varietyThis subsection highlights the geometry of subvarietiesparameterizing rank deficient endomorphisms (up toscale) for a three-dimensional vector space V .The graph of a planar homography H is given by theset of pairs of corresponding points ( x i , x (cid:48) i ) ∈ P × P contained in two views modeled as projective planesfulfilling H ( x i ) = x (cid:48) i . From a global point of view, theambient space is given as the image of the Segre embed-ding s , : P × P (cid:44) → P , i.e. it is a 4-dimensional alge-braic variety given at each point by the intersection offour functionally independent quadrics[4]. As Im ( s , )parameterizes the set of bilinear relations between twoprojective planes, and each projective plane has a pro-jective reference given by 4 points, a general homog-raphy H can be described in terms of two 4-tuples ofpoints that can be re-interpreted as the eight (nine upto scale) projective parameters of a general matrix H . This justifies perturbation arguments or, alternately, itprevents against the indiscriminate use of linear interpolationfor a non-linear variety as the cubic hypersurface D .omplete Endomorphisms in Computer Vision 5 The space of (3 × X up to scale is a pro-jective space P , which is homogeneous by the actionof the projective linear group P GL (9). The projectivelinear group P GL (3) induces an action on P End ( V )that breaks the initial homogeneity of P due to therank stratification given by three orbits. Each orbit ischaracterized by the rank constancy of a representativematrix. In particular, if M r denotes the algebraic sub-variety of matrices (up to scale) of rank 1 ≤ r ≤
3, thenthere is a natural rank stratification M ⊂ M ⊂ M .Planar homographies may be viewed as elements of M \ M up to scale. This rank decomposition is re-stricted in a natural way to the fundamental variety F . This subsection includes some results regarding the en-domorphisms
End r ( V ) of a vector space V ⊂ R , where r denotes the rank of a generic element. The first resultprovides a description of endomorphisms End ( V ) andits singular locus corresponding to degenerate endomor-phisms End ( V ). The second result gives its structureas a locally homogeneous space, i.e. as a disjoint unionof G -orbits by the action of GL (3) on the vector spaceof g . As usual, their elements are regular or eventuallydegenerate matrices, but their meaning is different asLie group or Lie algebra, respectively. Proposition 1
For any three-dimensional vector space V : a) the set of singular endomorphisms End ( V ) isa algebraic variety of codimension given by a cubichypersurface for n = 2 , which is a subregular orbit by the action of P GL (3) corresponding to “subregular”elements located in the adherence of the set of homo-graphies in P ; b) its singular locus is given by rank End ( V ) , which is a codi-mension manifold (smooth subvariety) diffeomorphi-cally equivalent to P Proof a) It is proved taking into account the character-ization of singular endomorphisms by the vanishing ofthe determinant of a generic 3 × P , its singular locus is locally de-scribed by the vanishing of determinants of all (2 × A representing any endo-morphism up to scale. Using ( a ij ) as local coordinatesin P , if a (cid:54) = 0, then a local system of independentequations (local generators for the ideal of the determi- nantal variety representing rank 1 matrices) is locallygiven by a − a a a − a a a − a a a − a a (1)in the open coordinate set D + ( a ) := { a ∈ P | a (cid:54) =0 } of P . They are functionally independent (i.e. its ja-cobian matrix has maximal rank) between them. Hence,they define a smooth variety of codimension 4 (differ-ential map of the above equations has maximal rank),which is locally diffeomorphic to P . The induced groupaction allows to extend the local diffeomorphism to aglobal diffeomorphism. In particular, it is locally pa-rameterized by a , a , a , a corresponding to el-ements in the complementary box of a (obtained byeliminating the row and the column of a ).Formally, the involution on spaces that exchangessubindexes (fixed points for transposition) leaves invari-ant the first and fourth generators, and identifies thesecond and third generators between them. Such invo-lution corresponds to a representation of the symmetricgroup, giving the local generators for the Veronese va-riety of double lines, which is isomorphic to the dual( P ) ν counted twice.Anyway, the rank stratification can be reformulatedin homogeneous coordinates as follows: Corollary 1
The action of P GL (3) on P End ( V ) givesan equivariant decomposition in three orbits charac-terized by the rank of the representative matrix up toscale. In particular: 1) the set of rank 1 endomorphisms End ( V ) (up to scale) is a 4D smooth manifold whoseprojectivization is diffeomorphic to P , which is a closedorbit by the induced action; 2) the set of rank 2 endo-morphisms End ( V ) (up to scale) is a 7D subregularorbit ; and 3) the set of homographies corresponding toregular endomorphisms (up to scale) End ( V ) is the regular orbit . The stratification of endomorphisms up to scale in-volving the projective model of planar views can begeometrically reinterpreted by reconstructing the va-riety E of degenerate endomorphisms as the secantvariety Sec (1 , End ( V )) in P of the smooth manifold End ( V ). Secant varieties are explained in Section 3.1.2.The action of GL ( n +1) can be extended in a naturalway to the k -th exterior power involving ( k + 1)-tuplesof vectors and their transformations for 0 ≤ k ≤ n .Thus, a locally symmetric structure is obtained for ar-bitrary configurations of ( k + 1)-tuples of vectors (or k -tuples of points). It is extended in a natural way to Finat, Javier, Delgado-del-Hoyo, Francisco linear envelopes of ( k +1)-dimensional vector subspacesor, in the homogeneous case, to k -dimensional projec-tive subspaces giving linear envelopes for any geometricobject contained in the ambient space.The set of ( k +1)-dimensional linear subspaces W k +1 are elements of a Grassmann manifold Grass ( k + 1 , n +1); its projective version is denoted as Gr k ( P n ). Grass-mann manifolds are a natural extension of projectivespaces. They also provide non-trivial “examples” forhomogeneous spaces and their generalization to sym-metric spaces or spherical varieties, jointly with super-imposed universal structures (fiber bundles). They havebeen overlooked over the years despite the presence ofthe analysis based on subspaces in a lot of tasks. A briefintroduction to Grassmannian manifolds and their ap-plications is provided by [27]. Homographies, fundamental or essential matrices canbe viewed as PL-uniparametric families (linear pencils)of matrices that can be represented by secant lines. Sim-ilarly, secant planes would correspond to PL-biparametricfamilies (linear nets) of matrices, and so on. Additionalformalism is required for a systematic treatment of thesefamilies.Secant varieties provide a PL-approach to any vari-ety X relative to any immersion f : X → P N . They aregiven as the closure Sec ( k, X ) of points z ∈ L k ⊂ P N where L k = < x , . . . , x k > with x , . . . , x k ∈ f ( X ) arelinearly independent. The k -th secant map associatesto each collection of k + 1 l.i. points x , . . . , x k theirlinear span L k = < x , . . . , x k > . The closure of thegraph is called the secant incidence variety . The projec-tion of the last component on the Grassmann manifold Grass k ( P N ) = Grass ( k + 1 , N + 1) is called the k -thsecant variety (of secant k -dimensional subspaces) of X and it is denoted by S k ( X ) as subvariety of G k ( P N ).Since ordinary incidence conditions p ∈ L are in-variant by the action of the projective group, secant in-cidence varieties represent projectively invariant condi-tions too. These conditions extend the well-known tan-gency conditions z ∈ T x Y . Next, we provide a classicaldefinition for smooth manifolds: Definition 1
For any embedding M m (cid:44) → P N of a con-nected regular m -dimensional manifold M , the secantvariety Sec (1 , M ) (also called “chordal variety” in theold terminology) is defined by the closure of points z ∈ P N lying on lines xy (called “chords”, also), where( x, y ) ∈ M × M − ∆ M are different points belongingto M , where ∆ M := { ( x, y ) ∈ M × M | x = y } is thediagonal of M × M . Obviously, if 2 m ≥ N the secant variety Sec (1 , M )fills out the ambient projective space. More generally,the following result is true: Lemma 1 If M is a m -dimensional smooth connectedmanifold, the expected dimension of Sec (1 , M ) is equalto min (2 m + 1 , N ) . The lemma is a consequence of a computation ofparameters on connected smooth varieties. The dimen-sion of the secant variety can be lower, but exceptionsare well-known for a specific type of low-dimensionalvarieties called Severi varieties[28]. In particular, thechordal variety of the m -dimensional Veronese varietyhas dimension 2 m , instead of the expected dimension2 m + 1, providing the first non-trivial example of a Sev-eri variety. More generally, if M is a m -dimensional con-nected manifold and 2 m +1 ≥ N , as Sec (1 , M ) is a con-nected variety, then Sec (1 , M ) = P N (see [22, Page 40]for more details about the Veronese embedding).An alternative description for a secant variety canbe provided in a purely topological way. Let define thediagonal of the product M × M as ∆ M , i.e. the setof pairs ( x, y ) ∈ M × M such that x = y . If M isa smooth m -dimensional variety, then M is diffeomor-phic to ∆ M through the diagonal embedding. Hence,the normal bundle N ∆ M is isomorphic to the tangentbundle τ M . Note that τ M × M = τ M ⊕ τ M . This topolog-ical description is useful to detect “regular” directionsthorugh the singular locus:Each pair ( x , y ) ∈ M × M − ∆ M can be mapped tothe line (cid:96) = < x , y > (also denoted as x × y ). This mapdefines a morphism σ : M × M − ∆ M → Grass ( P N )called the secant map of lines. The closure of its imagecontains the set of tangent lines to M , which correspondto limit positions of secant lines when x , y coalesce inone point ( x , x ) ∈ ∆ M .The closure of the graph Γ σ of the secant map oflines is a (2 m + 1)-dimensional incidence smooth pro-jective variety in the product P N × P N × Grass ( P N )whose projection on last component is, by definition,the 1-secant (2 m + 1)-dimensional subvariety S ( M ) of Grass ( P N ). The tangent space at each point x ∈ M can be described as the set of tangent lines to curveshaving a contact of order 2 with x . This definition isformally extended to the singular case by taking localderivations. However, we use a more simplistic approachbased on the continuity arguments and a simple compu-tation of parameters, which gives the following result: Proposition 2
The incidence variety of secant linescontains the m -dimensional space T M of the tangentbundle τ M of the manifold M . Its projection S ( M ) ⊂ Grass ( P N ) omplete Endomorphisms in Computer Vision 7 on the image of Sec (1 , M ) by the secant map σ is a ( m + 1) -dimensional subvariety of the Grasmannian oflines. These descriptions show how secant lines can be un-derstood in terms of the geometry of the ambient pro-jective space or, alternately, in terms of the geometry ofGrassmannians of lines G ( P n ). The arguments are ex-tended to higher dimension and singular varieties in [6].Furthermore, they correspond to decomposable tensorswhich are useful for estimation issues, also. Results in previous subsection allow to manage degen-eracies in regular transformations and to perform a PL-control in terms of limit positions of secant varieties. Itsextension to singular cases requires also the followingresult:
Proposition 3
Let denote the three-dimensional alge-braic variety of degenerate rank fundamental matricesas F , then Sec (1 , F ) = F and Sec (1 , F ) = P .Proof It suffices to prove that if rank ( F ) = rank ( F (cid:48) ) =1, then rank ( F + λ F (cid:48) ) ≤
2, i.e. | F + λ F (cid:48) | = 0. So, letdefine F = ( f f f ), where f i is the i -th column of thematrix F for 1 ≤ i ≤
3. Then, | F + λ F (cid:48) | is computed asthe arithmetic sum (up to sign) of determinants whichare always null. More explicitly, | F + λ F (cid:48) | = | f f f | + λ ( | f f f (cid:48) | + | f f (cid:48) f | + | f f f | )+ λ ( | f f (cid:48) f (cid:48) | + | f (cid:48) f f | + | f (cid:48) f (cid:48) f | )+ λ | f (cid:48) f (cid:48) f (cid:48) | The first and last summand vanish since F , F (cid:48) ∈ F .By developing each determinant by the elements of thecolumn (Laplace) and by using that all 2 × F and F (cid:48) vanish the proposition is proved.This result can be also applied to symmetric ma-trices up to scale in the projective space P of planeconics: Corollary 2
Let denote the two-dimensional algebraicvariety of degenerate rank symmetric matrices as Q ,then Sec (1 , Q ) = Q and Sec (1 , Q ) = Q = P .Proof The second equation is trivial by connectednessproperties and dimensional reasons. An intuitive prooffor the first equation is obtained from Proposition 3by the involution that exchanges subindex coordinates.Intuitively, the generic element of any pencil generatedby two double lines is a pair of intersecting lines[28]. The extension of the Corollary 2 to the space P ofquadrics in P is meaningful for 3D reconstruction is-sues. Indeed, a projective compactification of essentialmanifold E is isomorphic to a hyperplane section of thevariety of quadrics in P of rank r ≤ degenerate fundamentalmatrix F ∈ F , a generic segment F + λ F (cid:48) connect-ing two rank 1 fundamental matrices gives a genericrank 2 fundamental matrix. As consequence, a genericperturbation with any PL-path removes the indeter-minacy, and recovers a generic rank 2 fundamental ma-trix. This perturbation method (valid for stratificationswith “good incidence properties” for adjacent strata)provides a structural connection between fundamentalmatrices and homographies, which can be extended toessential matrices (see Section 4.1.4).3.2 Removing indeterminaciesThere are different kinds of indeterminacies for mul-tilinear approaches including structural relations be-tween pairs (given by fundamental and essential matri-ces, typically) or triplets of views (trifocal tensors, e.g.).All of them can be interpreted as singularities of the va-riety of corresponding tensors, including the fundamen-tal variety F or the essential variety E of P End ( V ).Indeterminacies can be removed using secant lines tothe singular locus F := Sing ( F ) of F (introduced inSection 3.1) or more generally k -th exterior powers.Secant lines allows to control degenerated matricesusing chords cutting singular varieties. These chordsinterpolate between more degenerate cases to recover avalid case. This subsection explains an alternative re-covery strategy based on the “recent history” of tangentvectors to the camera poses. This also requires a carefulanalysis of tangent spaces to matrices in terms of theiradjoint matrices. A regular transformation can be described by a matrix A with non-vanishing determinant or, from a dual view-point, by the adjoint matrix adj ( A ). Entries A ij of theadjoint matrix are the adjoint of each element a ij ∈ A up to scale; in the regular case, one must multiply with det ( A ) − . Hence, entries of the adjoint matrix repre-sent up to scale the gradient field of the array a linkedto A . Finat, Javier, Delgado-del-Hoyo, Francisco
For non-regular matrices (i.e. matrices A with van-ishing determinant) both descriptions are no longer equiv-alent between them. Let study the simplest case for n = 3 where the adjoint can be extended to non-regularmatrices. In the regular case, the adjoint is formally de-scribed as the second exterior power (cid:86) A of A . Theaction of GL (3 , R ) = Aut ( V ) on End ( V ) = g inducesan action of the second exterior power on the spaceof 2-dimensional subspaces (bivectors), whose projec-tivization represents lines of the projective plane as-sociated to the image plane. Thus, the Adjoint map adj : A (cid:55)→ adj ( A ) replaces the study of loci character-ized by 0 D features by their dual 1 D features supportedby projective lines. Loci and enveloping hyperplanes areequivalent between them for regular matrices.This naive approach has some implications to re-move indeterminacy when rank ( F ) = 1). In order tounderstand them F must be replaced by an enlargedspace that includes the different ways of approachingeach element F ∈ M by an “exceptional divisor”. Eachdivisor is supported by a finite collection of hypersur-faces in the ambient space M representing different ap-proaching ways to the singular locus, including elementsof M . This process is known in Algebraic Geometry asa blowing-up or σ -process [22]. An almost-trivial example
Bilinear maps on a vectorspace are represented by a 4-dimensional arbitrary spacecoupled with an inner product < X, Y > = tr ( XY ). Adirect computation shows that an orthogonal basis with T r ( X i ) (cid:54) = 0 and T r ( X i X j ) = 0 is given by (cid:18) (cid:19) , (cid:18) (cid:19) , (cid:18) (cid:19) , (cid:18) − (cid:19) Note that X , X and X + X are second ordernilpotent operators and that the last generator X pro-vides the bilinear structural constraint for the symplec-tic geometry on V [16,8]. In this case, ad ( X ) = X , ad ( X ) = X , ad ( X ) = − X , and ad ( X ) = X , with det ( X ) = det ( X ) = 0, det ( X ) = − det ( X ) =1. From a projective viewpoint, the set of bilinear mapson the projective line P can be reinterpreted in termsof the image of the Segre embedding s , : P × P → P given by([ x : x ] , [ y , y ]) (cid:55)→ [ x y : x y : x y : x y ]Let be z ij := x i y j then z z − z z = 0 defines aone-sheet hyperboloid in P . Hence, the set of bilinearmaps on V (up to scale) can be modelled with suchhyperboloid. The main novelty here concerns the rul-ing given by the group action on the projective lines (cid:96) = P < X , X > and (cid:96) = P < X , X > . In thiscase, the adjoint map induces an involution that canbe translated to a complex conjugation between thegenerators for each ruling. From a local topological viewpoint, degenerate endo-morphisms can be studied in terms of limits of tangentvectors X = t A γ ( t ) to curves γ ( t ) through A . Suchcurves represent trajectories in the matrix space con-necting “consecutive” poses for a camera. Regularity ofgeneric points of such curves allows to define tangentvectors at isolated degeneracies by means secant lines.A more intrinsic approach to tangency conditionsmust include the dual matrix for any X ∈ End ( V ). Itis defined at each point by the adjoint matrix adj ( X )whose entries X ij are the signed determinants of com-plementary minors of x ij . If X is a regular matrix, then X .adj ( X ) is a power of det ( X ) (cid:54) = 0, i.e. a unit froma projective viewpoint. We are interested in extendingthis construction to singular endomorphisms by usingintermediate exterior powers. Their closure in the corre-sponding projective space forms the variety of completeendomorphisms.To begin with, a first order complete endomorphismrepresenting a degenerate planar transformation is given,up to conjugation, by a pair of matrices ( X , X ν ), where X is an endomorphism of a 3D vector space V , and X ν represents its dual given up to scale by the ad-joint matrix adj ( X ) = (cid:86) X . The replacement of amatrix with its adjoint transforms any incidence con-dition (pass through a point for a conic, e.g.) into atangency condition (dual line becomes tangent at apoint, e.g.). Moreover, this exchange between projec-tively invariant conditions does not depend of the di-mension. For regular homographies, there exists a nat-ural duality between descriptions in terms of the origi-nal matrix A and its adjoint matrix adj ( A ), giving thenatural duality between incidence and tangency con-ditions for smooth “objects” (endomorphisms, in ourcase). Hence, the only novelty appears linked to singu-lar strata which can be illustrated by its applicationto the variety of singular fundamental matrices when n = 3. The dual construction is compatible with the in-duced action of P GL (3) on P End ( V ) and its restrictionto F . Let X ∈ g , g = T e G , then X ν ∈ g ∗ .For arbitrary dimension, the entries X ij of the ad-joint matrix adj ( X ) = (cid:86) n X = det ( X ) for X ∈ End ( V n )are interpreted as determinants corresponding to thecomponents of ∇ det ( X ). The next iteration for X ij gives the determinants of ( n − × ( n − ∇ X ij . Their vanishing defines omplete Endomorphisms in Computer Vision 9 the singular locus of the variety det ( X ) = 0. Symbol-ically, ∇ det ( X ) can be formulated as a double itera-tion of the adjoint map generated by the vanishing ofdeterminants of minors of size n − X . These deter-minants are the generators of (cid:86) n − End ( V n +1 ), whichis the dual of (cid:86) End ( V n +1 ).The description of the previous paragraph can begeometrically reinterpreted in terms of arbitrary codi-mension k subspaces. In particular, the extension of theadjoint map can be algebraically interpreted as a gra-dient field. For any endomorphism X let consider the n -tuples( X , (cid:94) X , . . . , n (cid:94) X ) , (2)where (cid:86) k X is the k -th exterior power of X , whoseentries are given by the determinants of the k × k -minorsof X . It is an element of the exterior algebra (cid:86) ∗ End ( V )defined by the direct sum of exterior powers of End ( V ).The iteration of the gradient field given by the deter-minant function det : End ( V ) → R as X (cid:55)→ det ( X )can be interpreted as “successive derivatives” on thespace of endomorphisms. Let us remark that traces ofexterior powers are the coefficients of the charatceristicpolynomila | λI − A | ;, which can be reinterpreted (inthe complex case) in terms of eigenvalues. Thus, in thiscase all the information is computable in terms of SVDwith usual interpretation for the ordered collection ofeigenvalues.In arbitrary dimension, the generic case correspondsto the regular orbit , i.e. endomorphisms X with rank ( X ) = n +1 (automorphisms). By iterating the construction ofexterior powers, one can associate an algebraic invariantgiven by the multirank rank ( (cid:86) k +1 X ) = rank ( (cid:86) n − k X ).Looking at Figure 2, the regular case for n = 2 (resp. n = 3) corresponds to bilinear forms with birank (3 , , ,
4) with self-duality for themid term), which can be reinterpreted in terms of quadraticforms. The case for non-regular orbits is constructedrecurrently: let k = corank ( X ) be the dimension of L k = ker ( X ), the indeterminacy is removed by addingthe complete bilinear as the linked quadratic forms on L k .For example, for any symmetric endomorphism X whose projectivization is a rank 1 plane conic, thereexists a double line (cid:96) = 0 whose kernel is the wholeline. The reduced 1 D kernel (cid:96) is also the support for a0 D conic on the line given by two different points (rank2) or a double point (rank 1), which define two orbitslabeled as (1 ,
2) and (1 ,
1) in Figure 2. Similarly, fora rank 1 quadric supported by a double plane L , thekernel is the whole plane that supports an embeddedcomplete conic ( q, q ν ) in the double plane with biranks (3 , , ,
2) and (1 , , , , , , ,
2) and (1 , , k ≥ k -dimensional tangent linear subspaces to“any object” contained in P n of increasing dimension.In this case, (a) the action of GL ( n + 1) induces a con-jugation action of (cid:86) k GL ( n + 1) on itself; (b) the k -thexterior powers of X can be considered as elements ofthe k -th exterior power of g ∗ .Essential manifold E of regular essential matrices isembedded in P ( End ( V )), a 15-dimensional projectivespace. The extension of complete homographies on P to P creates complete collineations ( X , (cid:86) X , (cid:86) X ).The third component (cid:86) X is in fact the adjoint matrixof X . This construction provides a general frameworkto obtain compactifications (as complete varieties) oforbifolds corresponding to F and E as degenerate en-domorphisms.The basic idea for extending regular to singular casesis based on adding infinitesimal information from suc-cessive adjoint maps, which is interpreted as the itera-tion of the gradient operator applied to the determinantof square submatrices. For generic singularities (i.e. forcorank c = 1) it suffices to replace the original formula-tion by its dual, which gives the tangent vector for smalldisplacements. For singularities with corank c ≥ c = 2 , n = 3. Adjacency relations between closures of orbits for rankstratification of
End ( V ) can be symbolically representedby the oriented graph of Figure 1. The vertices of thetriangle represent an orbit labeled with 3 , ,
1, accord-ing to the rank of the matrix representing the endomor-phism. Oriented edges are denoted as e , e and e and represent the following degeneracies: – →
2, corresponding to degeneracies of endomor-phisms to fundamental matrices. – →
1, corresponding to degeneracies from funda-mental to degenerate fundamental matrices. – →
1, corresponding to degeneracies from rank 3endomorphisms to degenerate fundamental matricesRight-side vertices of Figure 1 represent the birankscorresponding to the original matrix and its general-ized adjoint. A simple example is useful to illustrate (1,1)(1,2) (2,1)(3,3)1 23
Figure 1
Oriented graphs for the adjacency relations be-tween the closures of orbits for the rank stratification of
End ( V ). Nodes represent a rank stratification whilst edgesrepresent degeneracies in an orbit. the idea. Let suppose X ∈ End ( V ) as a diagonalizablematrix and λ , λ and λ as the eigenvalues of A , thenfundamental matrices have at least one vanishing eigen-value. In the complex case canonical diagonal formswould be equivalent to diagonal matrices ( λ , λ , λ )whose adjoint would be ( λ λ , λ λ , λ λ ), ( λ λ , , , , M must be replaced by the excep-tional divisor E of the blowing-up of P with center thesmooth manifold M . E has the orbits by the adjointaction as components, represented by the pairs (1 , , G replaces the oriented tri-angle whose vertices are labeled as 3 , G (1) . This graph is an oriented square whose ver-tices are labeled as (3 , , (2 , ,
2) and (1 , A , adj ( A )). Biranks are relative to the generic ele-ments of the topological closure for the orbits of theproduct action G × (cid:86) G , where G = Aut ( V ) = GL (3),on the space g × (cid:86) g , where g = End ( V ) = g (cid:96) (3),acting on the graph of the adjoint map.Hence, biranks encapsulate numerical invariants forthe natural extension of the action of the Lie alge-bra g (cid:96) (3) corresponding to End ( V ) = T I Aut ( V ), us-ing pairs of infinitesimal transformations on the origi-nal and dual spaces. The simultaneous management ofcomplete endomorphisms ( X , X ν ) is more suitable fromthe computational viewpoint. Indeed, a simple exten-sion of SVD methods to pairs of endomorphisms allowsan estimation of the generators of Lie algebras g andits dual g ∗ easier than the estimation of the generatorsfor the original Lie group G . 3.3 Complete fundamental matricesThis section adapts constructions of endomorphisms inany dimension shown in precedent section to fundamen-tal matrices. The notion of complete fundamental ma-trix is crucial to perform a control of degenerate casesfrom a quasi-static approach. This means that we arenot taken into account the kinematics of the camera.Nevertheless, as the adjoint f a represents the gradientvector field ∇ f a F at the element f , there is a measure-ment of “local variation” around each element f ∈ F . Definition 2
The set of complete fundamental matri-ces is the closure F × F a of pairs ( F , F a ) ∈ where F ∈ F is a fundamental matrix of rank 2, and F a ∈ F a is its adjoint matrix whose entries are given by the de-terminants of all 2 × F .Degeneracies of endomorphisms up to scale linkedto camera poses can be controlled with the insertion oftangential information and the restriction of the con-struction to F . This information allows to represent“instantaneous” directions in terms of all possible di-rectional derivatives represented by the adjoint matrix.Thus, despite rank ( F ) = 2 and consequently rank ( F a ) =1, the adjoint matrix provides a parameterized sup-port (in fact a variety) to represent “directions” alongwhich the degeneracy occurs. However, this remark isnot longer true for the most degenerate cases corre-sponding to rank 1 matrices that require a set of pairsof degenerate fundamental matrices. In the absence of perspective models supporting arbi-trary homographies, fundamental matrices are used toavoid indeterminacies. For consecutive camera poses,two fundamental matrices F (for views V and V ), and F (cid:48) (for views V (cid:48) and V (cid:48) ) provide a structural relationbetween both views.The dual construction X (cid:55)→ adj ( X ) correspondingto the gradient ∇ at each point can be restricted just torank 2 fundamental matrices. This provides a tangentialdescription with information about the first order evo-lution of F according to tangential constraints. From amore practical viewpoint, tangential information can beapproached by secant lines in a PL-approach that con-nects rank 2 fundamental matrices f , f (cid:48) ∈ F . However,this construction becomes ill-defined when rank ( F ) = 1since the adjoint map is identically null. Then, tan-gential directions corresponding to endomorphisms of W = Ker ( X ) are added to avoid this indeterminacy.For regular matrices the extended approach for com-plete objects is compatible with the differential approach omplete Endomorphisms in Computer Vision 11 given by a smooth interpolation along a geodesic path γ i,i +1 connecting consecutive complete fundamental ma-trices ( f i , f ai ) and ( f i +1 , f ai +1 ). These are obtained as thelift of a geodesic path to the tangent space. From atheoretical viewpoint, lifting is performed by restrict-ing the logarithm map. In practice, secant lines providea first order approach to geodesics.A more detailed study of the geometry of degener-ate fundamental matrices is required to recover a well-defined limit of tangent spaces in the singular case.This study must include procedures for selecting a PL-path (supported on chords) connecting the degenerated f (cid:48) with its neighboring generic fundamental matrices.In practice, if the sampling rate is high enough therewill be no meaningful difference between pairs of ma-trices. This generates uncertainty about the directionto approach the tangent vector. It can be solved with acoarser sampling rate along the “precedent story”. A local neighborhood of F ∈ F relative to F is theunit sphere corresponding to the fiber of the puncturednormal bundle of F in F . Normal bundles are givenby a quotient of tangent bundles. Hence, tangent bun-dles of F and F = Sec (1 , F ) must be computed. Thelatter can be described as the blow-up of F × F withcenter in the diagonal ∆ F . This diagonal is isomor-phic to the smooth manifold F so its tangent bundleis also isomorphic to the tangent bundle of F . Hence,it suffices to compute the latter and reinterpret it ingeometric terms.Note that F is a smooth manifold diffeomorphicto P , so their tangent bundles are isomorphic. Thus, τ F (cid:39) τ P can be reinterpreted in terms of incidencevarieties. Simplest incidence varieties in the projectiveplane P are given by I , := { ( p , (cid:96) ) ∈ P × ( P ) ν | p ∈ (cid:96) } . There are two projections on P and ( P ) ν to in-terpret the incidence variety as the canonical bundleof the projective plane. This elementary constructionis extended to subspaces of any dimension k with thecanonical bundle on the Grassmannian of k -dimensionalsubspaces. An example is the incidence variety I , := { ( a , (cid:96) ) ∈ P × Gr ( P ) | a ∈ (cid:96) } , where Gr ( P ) denotesthe Grassmannian of lines in P .The restriction to the fundamental variety F of thesecond projection on Gr ( P ) gives the secant variety Sec ( F ) to F that fills out all the ambient projectivespace. In other words, along each point F ∈ P pass asecant line to F . The same is also true for E instead of F . Furthermore, an interesting result can be formulatedusing the same notation: Proposition 4
The D secant variety Sec ( F ) is agenerically triple covering of F that ramifies along the -dimensional tangent variety T ( F ) (total space of thetangent bundle) corresponding to tangent lines having adouble contact at each element F ∈ F .3.3.3 Triplets of fundamental matrices This Section is intended to provide an algebraic visu-alization of neighboring matrices at the most degener-ate case, which avoids the indeterminacy at rank 1 ele-ments. We use a geometric interpretation of the blowing-up process explained in [22] in terms of secant varieties.In our case, the blowing-up of the variety F re-places each rank 1 degenerate fundamental matrix F ∈F with a 3D subspace generated by four linearly in-dependent vectors. They can be interpreted in terms ofsecant lines connecting the point f for matrix F withfour independent points belonging to the subregularorbit F \F . Hence, each extended face of a generic“tetrahedral configuration” (see Figure 2) represents a3D secant projective space to F displaying degeneraciesat each F ∈ F .The cubic hypersurface representing fundamentalmatrices in P is not a ruled variety. Thus, a generictriplet of fundamental matrices generates a 2-dimensionalsecant plane to the hypersurface det ( F ) = 0. Obvi-ously, the variety of secant lines to F and trisecant2-planes to F fills out the projective space F = P of endomorphisms up to scale. More specifically, anyhomography can be expressed as a linear combinationof three generic fundamental matrices (similarly for es-sential matrices).The nearest 3-secant 2-plane < F , F , F > foreach rank 1 fundamental matrix F ∈ F can be com-puted using a metric on the Grassmannian of 2-planes.The most common metric is the inner product < X , Y > = tr ( X . Y )of g = End ( V ). In particular, at each degenerate en-domorphism X corresponding to a fundamental matrix F (rep. essential matrix E ), two different eigenvalues λ i , λ i (resp. a non-zero double eigenvalue λ i ) are ob-tained for 1 ≤ i ≤ F i with maximal dis-tance in the plane ( λ , λ ) of non-null eigenvalues. Thisdistance is determined w.r.t. the eigenvalues ( λ,
0) or(0 , λ ) of the degenerate fundamental matrix F (simi-larly for the essential matrix). The application of thistheoretical remark would must allow to escape fromdegenerate situations to avoid collisions against pla-nar surfaces (corresponding to walls, floor, ground, e.g.) whose elements do not impose linearly independent con-ditions to determine F . From a more practical view-point, the problem is the design of a control device ableof identifying the “best” escape path in a continuousway, i.e. without applying switching procedures. In thenext paragraph we give some insight about this issue. This section explains how complete matrices, which canbe read in terms of successive envelopes, provide specificcontrol mechanisms to avoid degeneracies appearing inrank 1 fundamental or essential matrices.A basic strategy to analyze and solve the indetermi-nacy locus of an endomorphism consists in augmentingthe original endomorphism by their successive exteriorpowers. The properties of the adjoint matrix providea geometric interpretation in terms of successive gen-eralized secant envelopes by k -dimensional subspaces .Consecutive iteration on the adjoints can be viewed asa Taylor development so that when rank ( X ) decreases,all complete objects contained in ker ( X ) can be addedto remove degeneracies.The lifting of the action of Aut ( V ) on End ( V ) totheir k -th exterior powers delivers a structure as lo-cally symmetric spaces for the set of complete objectslinked to End ( V ). Here Aut ( V ) can be replaced witha group G , the dual of End ( V ) (corresponding to takeadjoint matrices) with g ∗ and the induced action bythe adjoint map ad : G → g ∗ giving the adjoint ac-tion ad : G × g ∗ → g ∗ . Then, the k -th exterior power (cid:86) k ad : (cid:86) k G → (cid:86) k g ∗ is the natural extension of theadjoint map. This map induces the corresponding k -thadjoint action of (cid:86) k G on (cid:86) k g ∗ that extends the origi-nal action of Aut ( V ) on End ( V ) ∗ . This simple construc-tion is applicable for all actions of classical groups toremove their possible indeterminacies on Lie algebras.The simplest non-trivial example is the pair ( k, n ) =(2 , (cid:0) (cid:1) for (cid:86) V . In this case, the 6 × (cid:86) G (automorphisms of (cid:86) V ) act on the 6 × (cid:86) k g ∗ (endomorphisms of (cid:86) V ). In prac-tice it requires to compute the determinants of 2 × × × × , ,
2) of arbitrary 4 × , , × × This is valid for tangent subspaces as limits of secantsubspaces in the Grassmannian terms of ordinary rotations. In addition, this construc-tion can be adapted to bilateral (product or contact)actions in terms of double conjugacy classes. We are in-terested in the locally symmetric structure of the rank-stratified set of projection matrices linking scene andviews models. A simple description of this structure forany space allows to propagate control strategies by us-ing local symmetries, without using differential meth-ods, no longer valid in the presence of singularities.The completion of planar homographies in P to in-clude fundamental matrices can be extended to any di-mension and re-interpreted in matrix terms. To achievethis goal, it is required to consider the projectivization P ( End ( V n +1 )) of a ( n +1)-dimensional space V , to con-struct the rank stratification of matrices in End ( V ) andto take the ( k + 1)-th exterior power (up to scale) ofthem for 0 ≤ k ≤ n . The locally symmetric structurefor the resulting completed space is obtained by the in-duced action of GL ( n + 1) = Aut ( V n +1 ) on the ( k + 1)-th exterior power (cid:86) k +1 V of V . This structure justifiespositional arguments for minimal collections of corre-sponding elements (points, lines, or more generally, lin-ear subspaces) and controls their possible degeneraciesin terms of adjacent orbits.The construction of the above completion poses somechallenges. For example, its topological description in P N for N = ( n + 1) − k + 1)-dimensional linear subspaces in the ambient projectivespace P n . Using contact constraints for linear subspaceshas an equivalence to scene objects in terms of PL-envelopes by successive higher dimensional linear sub-spaces L k +1 . Due to space limitations, we constrain our-selves to the case n = 3 and the completion of essentialmatrices.The fact that End ( V ) (including degeneracies) isthe Lie algebra of Aut ( V ) (only regular transforma-tions) links algebraic with differential aspects. Hence,exponential and logarithm maps provide a natural re-lation between both of them. However, as fundamentaland essential matrices play a similar role for affine andeuclidean frameworks (as non-degenerate bilinear rela-tions), a common framework where both interpretationsare compatible is required for an unified treatment ofdegenerated cases.4.1 Essential manifoldThe essential constraint for pairs ( p , p (cid:48) ) of correspond-ing points from a calibrated camera is given by T pEp =0. The set of essential matrices E is globally character-ized as a determinantal variety in [11, Section 2.2]. Theauthor explores the structure of E as a locally symmet-ric variety and a completion (not necessarily unique) omplete Endomorphisms in Computer Vision 13 obtained using elementary properties of adjoint matri-ces.Any ordinary essential matrix E has a decompo-sition E = RS , where R is a rotation matrix and S is the skew-symmetric matrix of a translation vec-tor t . Essential and fundamental matrices are relatedthrough E = M Tr FM (cid:96) where M r (resp. M (cid:96) ) is anaffine transformation acting on right (resp. left) on thesource (resp. target space). In algebraic terms, they be-long to the same double conjugacy class by the diagonalof the A -action of two copies of the affine group, where A = R × L is the direct product of right R and left L actions. Hence, essential matrices can be consideredas equivalence classes of fundamental matrices. The fol-lowing result gives a synthesis of the above considera-tions: Proposition 5
The variety E i of extended essential ma-trices of rank ≤ i is a quotient of the variety F i ofextended fundamental matrices of rank ≤ i . More gen-erally, the stratified map F → E is an equivariant fi-bration between stratified analytic varieties for naturalrank stratifications in P End ( V ) . The exchange between projective, affine and euclideaninformation and the analysis of degenerate situationsrequires a general framework where rank transitions canbe controlled in simple terms. To accomplish this goal,a locally symmetric framework that represents degen-erate cases must be developed.Degeneracies can be studied in the space of multilin-ear relations between corresponding points, from whichthe essential matrix is estimated [12]. However, these es-timations are performed in the space of configurationsof points without considering the degeneracies of thematrices in the space of endomorphisms arising froman algebraic viewpoint. The secant line connecting twodegenerate endomorphisms is translated in 8+8+1 = 17linear parameters in the same way as in [12]. Additionalconstraints relative to fundamental or essential matri-ces help to reduce the number of parameters.If X is a variety with singular locus Sing ( X ), theregular locus is denoted as X := X \ Sing ( X ). In par-ticular, the set of regular fundamental matrices is de-noted by F = F \F and the set of essential matri-ces as E = E \E . Rank stratification of End ( V ) is F ⊂ F (resp. E ⊂ E ), where the subindex k denotesthe algebraic subvariety of matrices with rank ≤ k , upto scale .Two global algebraic and differential results to con-sider are the following ones: Fundamental matrices are considered in Section 3.1. – The essential variety E is a 5-dimensional degree 10subvariety of P , which is isomorphic to a hyper-plane section of the variety V of complex symmet-ric matrices of rank ≤ P via the degree 2 Veronese embed-ding v , whose image is the variety V of doubleplanes. In particular Sec (1 , V ) = V . – If T SO (3) (cid:39) SO (3) × R (cid:39) SO (3) × so (3) denotesthe total space of the tangent bundle of SO (3), thenthe essential regular manifold E is isomorphic tothe total space of unit tangent bundle of SO (3)given by SO (3) × S .Nevertheless their local description in terms of Liealgebras, all the above isomorphisms are global sinceany Lie group G is a parallelizable manifold. If G isconnected, the isomorphisms are infinitesimally givenby the translation T A G = A. g . In the euclidean frame-work, this description allows to decouple rotations andtranslations. The rest of this Section explains local andglobal properties of extended E and their relations withfundamental matrices. An essential matrix E ∈ E is a 2-rank matrix withtwo equal eigenvalues and a diagonal form ( λ, λ, , , decomposes E in a prod-uct U ΣV T where U, V are orthogonal and det ( U ) =+1 = det ( V ). The sign of the determinant can be cho-sen without modifying the SVD. Then, the fibration Φ : SO (3) × SO (3) → E given by Φ ( U, V ) =
U ΣV T is a submersion with a 1-dimensional kernel represent-ing the ambiguity for choosing the basis of the spacegenerated by the first two columns of U and V .The description of E in terms of the fibration Φ al-lows to decompose any element E ∈ E in “horizontal”and “vertical” components for the tangent space to theproduct SO (3) × SO (3). This is crucial to reinterpretthe decomposition in locally symmetric terms and tobound errors linked to large baselines[23]. The set E of regular essential matrices is an open man-ifold that can be globally described in terms of the unittangent bundle τ u SO (3) to the special orthogonal group SO (3) representing spatial rotations[18]. Also, o (3) := T I SO (3) is the Lie algebra of SO (3), i.e. the vectorspace of skew-symmetric 3 × T A SO (3) = Singular Value Decomposition4 Finat, Javier, Delgado-del-Hoyo, Francisco A. so (3) at each A ∈ SO (3). Then, E is a 5-dimensionalalgebraic variety contained in the total space T u SO (3) (cid:39) SO (3) × S of the unit tangent bundle τ u SO (3). In thisbundle each fiber takes only unit tangent vectors so werestrict to unit vectors X ∈ so (3).The isomorphism S = SO (3) /SO (2) enables a rein-terpretation of the unit vectors as spatial rotations mod-ulo planar rotations. Each element determines a uniquerotation axis, where the rotation through angles θ + π and θ − π are identical. Hence, SO (3) is homeomorphicto RP , which provides a general framework for a pro-jective interpretation in terms of space lines (see [18,Section 3.2]). Inversely, euclidean reduction of projec-tive information can be viewed as a group reductionto fix the absolute quadric that plays the role of non-degenerate metric (see Section 4.1.3). The projective ambiguity of projective lines as rota-tion axis gives two possible solutions for correspondingelements of SO (3) [3, Section 3]. This ambiguity canbe modelled as a reflection that exchanges the currentphase with the opposite phase between them. Hence,pairs of regular essential matrices in SO (3) × SO (3),where the the second component is generated by thelogarithmic map, generate a quadruple ambiguity cor-responding to two simultaneous reflections.Using the topological equivalence between SO (3)and RP , the ambiguity can be represented by the prod-uct of two copies of the tangent bundle of the projectivespace where elements ( x , v ) and ( − x , − v ) are identi-fied by the antipodal map. This natural identificationof the tangent bundle τ RP has not a kinematic mean-ing from the viewpoint of the “recent history”. Hence,in order to solve the ambiguity, a C -constraint must beinserted into the essential matrices for precedent cam-era poses. However, this constraint is only valid undernon-degeneracy conditions for essential matrices, i.e.the eigenvalue λ must be non-null. Otherwise, essen-tial matrix “vanishes” and it cannot be recovered. The manifold E of regular essential matrices can bevisualized as a smooth submanifold of spatial homogra-phies P GL (4) given as the open set of regular transfor-mations as points of P . These regular transformationsare described by automorphisms A ∈ Aut ( V ) of a 4-dimensional vector space V whose Lie algebra is givenby End ( V ), including possible degeneracies. Our aimis to study these degenerate cases by using locally sym-metric properties extending T R SO (3) = R so (3). Metric distortions must be avoided when approaching to thesingular locus.The vector space End ( V ) of 4 × X hasa natural stratification by the rank rank ( X ) denotedby W ⊂ W ⊂ W ⊂ W . Here, W i represents the al-gebraic variety defined by the vanishing of all determi-nants of size ( i + 1) × ( i + 1), which are endomorphismsof rank ≤ i . More concretely, ordinary homographies A ∈ P GL (4) are represented by points of the open set W \ W . Obviously, W is the natural generalization offundamental matrices given by det ( X ) = 0.In this framework, the variety of secant lines Sec ( E )to E (cid:39) SO (3) × S in P is a 11-dimensional projectivevariety. Similarly, the variety of secant planes Sec ( E )to E fills out the ambient space P . This means thatany spatial homography of P can be described by threeessential matrices, but not in a unique way. Even more,secant varieties to rank-stratified varieties of projectiveendomorphisms can be described in terms of locallysymmetric varieties. The simplest case corresponds tosymmetric endomorphisms representing eventually de-generate conics or quadrics.In particular, Figure 2 illustrates how to extend Fig-ure 1 to the third dimension for the symmetric case.These orbits are induced by the action of P GL (4; C )on complete symmetric complex endomorphisms. Thefirst blow-up in the graph replaces the vertex labeledas 1 with the opposite face (with the same orientation).This vertex represents the endomorphisms of rank 1 upto scale or E , They are completed in the new face rep-resenting the three orbits of the variety E of 1-secants Sec (1 , E ). Next, the second blow-up at each minimalvertex at each height replaces such vertex by the oppo-site side in the height with the same orientation. Theresult is an oriented cubical 3D graph whose verticesrepresent orbits by the induced action of GL on suc-cessive exterior powers. Vertices are labeled accordingto the multirank of each symmetric endomorphism andtheir respective envelopes by linear subspaces.4.2 The adjoint representationAny element of a Lie group G can be lifted to its Lie al-gebra g with the log map log : G → g , the local inverseof the exponential map exp : g → G . The adjoint repre-sentation G → Aut ( G ) models G as a matrix group interms of its conjugation automorphism Φ g defined by Φ g ( h ) := ghg − for every g ∈ G . From the differentialviewpoint, the adjoint representation of G is computedas the differential map of Φ g at the identity e ∈ G , i.e. A : G → Aut ( g ) defined by A ( g ) := d e Φ g .Its dual is given by the coadjoint action K : G → Aut ( g ∗ ) defined by the adjoint representation of the in- omplete Endomorphisms in Computer Vision 15 (4) (2)(1)(3) (1,3) (1,1)(2,1)(3,3) (4,6)(1,2) (2,1,1)(3,3,1) (1,3,3)(1,1,1)(1,2,1) (1,1,2)(4,6,4) (2,1,2) Figure 2
Symbolic representation of the orbits generated bythe action of P GL (4). Dotted edges represent hidden edges.Grey-colored edges are new edges generated after a blow-up ofa vertex in a previous graph. The oriented tetrahedral graphgives the oriented triangular prism and this is finally con-verted into the oriented cube. verse element g − . Orbits by the coadjoint action sup-port a symplectic structure[13]. There are two meaningful examples of the adjoint rep-resentation for our case: – If G = GL (3) = Aut ( V ), where V is a 3D vectorspace, then g = End ( V ) is naturally stratified bythe rank. The coadjoint action corresponds to orbitsof adjoint matrices. Its compactification has beendescribed in Section 3.1. – If G = SO (3), then the adjoint representation is SO (3) → Aut ( so (3)). It maps each spatial rota-tion g ∈ SO (3) into the automorphism so (3) → so (3) representing an “infinitesimal” displacementbetween two consecutive poses as a translation inthe tangent space. The adjoint action can be inter-preted as the action induced by the differential ofthe Adjoint representation of G in Aut ( G ) given byordinary algebraic conjugation . A topological equivariant stratification of a G -space X w.r.t. a G -action G × X → X is a decomposition of X in a union of G -orbits, i.e. subsets of X invariant by The adjoint starts with uppercase when it refers to agroup. the action of G . The corresponding algebraic equivari-ant stratification corresponds to an algebraic action. Inthe presence of motion, it is interesting to construct asymplectic equivariant stratification to manage packs ofsolutions for structural motion equations [14]. This sub-section refers only to basic aspects of algebraic equiv-ariant stratifications.More formally, the compactification of the coadjointaction for GL ( n ) creates an equivariant decompositionof orbits for many general groups whose canonical formsare well-known (Jordan) [2]. Furthermore, this decom-position can be restricted to the coadjoint actions forany subgroup of G , such as SO ( n ) (preservation ofmetric properties), SL ( n ) (volume preservation, despiteshape changes), or Sp ( n ) (preservation of motion equa-tions).The above conservation laws w.r.t. the algebraic G -actions provide ideal theoretical structural constraints.The problem is solved by minimizing their infinitesi-mal variation in their Lie algebra space. The transfer-ence of information requires a more careful study ofalgebraic and differential relations, developed in Sec-tion 4.3. For the applications concerning this paper wehave constraint ourselves to only regular or subregularorbits.4.3 Relating algebraic and differential approachesThis subsection explores the relations between an equiv-ariant completion of the essential manifold and the fun-damental variety in the algebraic framework given bycoadjoint actions. More concretely, we provide descrip-tions of such completions in terms of locally symmetricspaces obtained by compactifying the original descrip-tions as homogeneous spaces. From a topological view-point, a compactification incorporates the behavior atboundaries of a topological space to preserve regularityconditions. These compactifications include additionalorbits corresponding to degeneracies in the topologicaladherence of regular orbits. Therefore, computations forthe regular case can be extended to degenerate casestoo. Essential matrices have two equal non-vanishing eigen-values and a null eigenvalue. A basic specialization prin-ciple suggests to obtain an essential matrix with a re-duction to the diagonal (on the space of eigenvalues),and the addition of constraints relative to SO (3) and itsLie algebra so (3). By the triviality of its tangent bundle, T R SO (3) is the translation R so (3) of the Lie algebra. Considering the degeneracies of fundamental and essen-tial matrices, it is convenient to visualize how topolog-ical arguments for regular points can be extended tosingularities. In particular, complete objects allow tocontrol degeneracies in the boundary of orbits and tominimize errors in the presence of such singularities.In affine geometry, a completion of degenerate fun-damental matrices allows to recover affine reconstruc-tions. Euclidean reconstructions (up to scale) can beachieved by preserving the absolute conic. However, eu-clidean information from the completion of degeneratefundamental matrices is more difficult to retrieve. A for-ward construction for fundamental matrices is not truebecause these matrices would become identically null.Thus, it is convenient to consider pairs (resp. triplets)corresponding to secant lines (resp. planes).The description of secant spaces as locally symmet-ric spaces requires some considerations about productsof algebraic or infinitesimal actions. The following sub-sections describe the topological case, the most generalone, with a basic distinction between the simplest prod-uct action and the contact action, which incorporatescoupling constraints.
Let
Homeom ( Z ) denote the group of homeomorphisms(bijective and bicontinuous maps) of Z such that for anycontinuous map f : X → Y between two topologicalspaces the A -action is defined by the direct or Cartesiangroup Homeom ( X ) × Homeom ( Y ). It acts on the spaceof maps as ( kf g − )( x ) = k ( f ( g − ) x ) for any x ∈ X and for any ( g, k ) ∈ Homeom ( X ) × Homeom ( Y ). Ob-viously, R -action on the source space X , and L -actionon the target space Y are two particular cases of theabove A -action.From a topological viewpoint, it is convenient torestrict the above topological action to the differen-tiable case using Dif f eom ( Z ) instead of Homeom ( Z )for Z = X or Z = Y . Homeomorphisms and diffeomor-phisms have been used in deformation problems [19].Since a global description of Dif f eom ( Z ) is complexand we are interested in the local aspects, only localdiffeomorphisms Dif f eom z ( Z ) are considered, i.e. dif-feomorphisms preserving a point z ∈ Z .The linearization of the differentiable A -action givesthe A := R × L -action on the differentiable map d x f : T x X → T y Y (locally represented by the Jacobian map J f ) at x ∈ X with y = f ( x ). If p = dim ( Y ), thenthe A -action is given by KJ f H − for any ( H, K ) ∈ GL ( n, R ) × GL ( p ; R ). When n = p = 3, the actionof GL can be reduced to the action of the orthogonalgroup O . Its restriction to the diagonal in O (3) × O (3) provides the relation between fundamental and essen-tial matrices.More generally, the matrix expression of the A -actionis given by the linearization of the double conjugacyclasses for the A -action. From the differential view-point, it can be written as the first order term of the1-jet j e,e A = A . This case comprises two linear actionsacting on the Jacobian matrix at right and left simulta-neously. In practice, the Lie algebras are preferred overthe Lie groups so instead of taking the direct productof Lie groups GL ( n, R ) × GL ( p ; R ), its infinitesimal ver-sion can be chosen. It is defined by the direct product g (cid:96) ( n ) × g (cid:96) ( p ) of their Lie algebras, which provide a uni-fied treatment including degenerate cases involving therestriction of endomorphisms to the corresponding Liealgebras g for each classical group G .Furthermore, this approach connects directly withthe extended adjoint representation from Section 4.2.These constructions can be adapted to any other classi-cal subgroup H ⊂ G , i.e. a closed subgroup preserving anon-degenerate quadratic, bilinear or multilinear form.This includes SO ( n ), SL ( n ), Sp ( n ), and similarly fortheir Lie algebras. In all cases, a specific G -equivariantdecomposition as union of G -orbits for the adjoint rep-resentation in the Lie algebra g is obtained. In the differentiable framework, contact equivalence pre-serves the graph Γ f of any transformation f : X → Y . Moreover, it introduces a natural coupling betweenactions on source X and target Y spaces for f , andconsequently for C r -equivalences acting simultaneouslyon X × Y . A meaningful example for our purposescorresponds to linear maps of central projections of abounded region of the image plane.Topological invariants for the transformation f : X → X are linked to the fixed locus of f [6]. Thislocus is the intersection product Γ f · ∆ X , which cor-responds to a weighted sum of pairs ( x, y ) ∈ X × X such that: a) y = f ( x ), i.e. they belong to the graph Γ f ; b) y = x , i.e. they belong to the diagonal ∆ X := { ( x, y ) ∈ X × X | x = y } ). Therefore, transformationrepresented by a group action can be computed frompairs of corresponding points.This argument can be extended to k -tuples of pointsrelated by transformations acting on source and targetspaces linked by the projection map. So, instead of re-moving 3D points generated from the pairs, which re-quires a posterior resolution of the ambiguity, they canbe saved for a low-level interpretation in the ambientspace. More formally, this ambiguity can be viewed asthe dependence loci for sections of a topological fibra- omplete Endomorphisms in Computer Vision 17 tion that takes values in the space of configurations.This idea is further explained in Section 5. This section is devoted to outline a procedure for an-ticipating changes in camera poses that can include de-generate cases too in order to provide some insight forthe corresponding control devices in autonomous navi-gation including degenerate cases. The key is to exploitthe locally symmetric structure of the space of projec-tion maps arising from the double action. Our strategyconsiders linear actions on source and target spaces de-scribed by endomorphisms. These actions can act in adecoupled way (left-right actions) or in a coupled way(contact action). Moreover, they can be described inalgebraic terms (using Lie groups) or in infinitesimalterms (using Lie algebras). The second approach allowsto incorporate degeneracies in a natural way, and de-velop “completion strategies” by using exterior powers,as described in Section 4.Eventual degeneracies in endomorphisms are man-aged by simultaneous actions on source and target spaces,which are subsets of the scene and views. Endomor-phisms can be completed using exterior powers to achievean equivariant stratification in terms of double conju-gacy classes. This results in the aforementioned struc-ture as a locally symmetric variety for the space of or-bits.5.1 Decoupled vs coupled actionsOur strategy comprises two steps. The first step consistin decoupling source spaces (completed by a projectivemodel for the 3D scene, e.g.) from target spaces (com-pleted by a projective model for each view) for maps.The second step incorporates a more realistic couplingbetween both spaces, which is natural since each viewis a projection of the scene.In the first step there is a decoupling between leftand right actions. This decoupling is firstly formulatedin algebraic terms, and next extended to infinitesimalterms with complete endomorphisms to include degen-erate cases. The main novelty w.r.t. precedent sectionsis the management of pairs of orbits involving sourceand target spaces. This structure can be adapted toany pair of classical groups related to the projective,affine or euclidean framework.The most regular algebraic transformations of thesource space (a three-dimensional projective space P = P V ) are defined by the group of collineations or, more specifically, spatial homographies P GL (4). Two mean-ingful subgroups are the affine group A G := GL (3) (cid:110)R (semidirect group of general linear group and thegroup of translations), and the euclidean group E G := SO (3) (cid:110) R (semidirect group of the special orthogonalgroup and the group of translations).A central projection P i : P → P with center C i is the conjugate of the standard projection ( I O ) bythe left-right action denoted by A := R × L . Here R = GL (4) (resp. L = GL (3)) acts on right (resp. left) bymatrix multiplication up to scale. Hence, description ofbasic algebraic invariants for projection maps must beposed using double actions on the space of maps. For any pair ( K , H ) of regular transformations, the ac-tion on any central projection matrix P i correspondingto P → P is defined by KP i H − up to scale. Thedouble conjugacy class of the (3 × P i for aregular central projection can be obtained by varying( K , H ) in GL (3) × GL (4).The simplest double actions in multiple view geom-etry are pairs of rigid motions or affine transformationsacting on source P and target P spaces for the centralprojection P i with center C i Proposition 6
Let define the double action by
R × L -action of pairs of diffeomorphisms acting on (germs of )maps f : R n → R p by double conjugacy, i.e. A f = { kf h − | for all ( k, h ) ∈ Dif f ( R n ) × Dif f ( R p ) } . Let ( h, k ) ∈ g (cid:96) ( n ) × g (cid:96) ( p ) be a pair of vector fields for ( K, H ) , then the tangent space to the left-right orbit A f is given by k ◦ f − df ◦ h .Proof By using the underlying topology of the set ofprojection matrices [ P ] ∈ P (up to scale), a “smallperturbation” P ε of the projection map P around anyelement x = { ( x ij ) | ≤ i ≤ , ≤ j ≤ j } represent-ing a 3 × P ε ( x ) = P ( x − εh ( P ( x )) + εk ( P ( x )) + . . . = P + ε [ k ( P ( x )) − ∂ P ∂ x h ( x )] + ε [ . . . ] + . . . Hence, the result is proved by taking limits in (cid:15) : lim ε → P ε ( x ) − P ( x ε = k ( P ( x )) − ∂ P ∂ x h ( x ) Remark:
This is a particular case of the descriptionof the tangent orbit to the A -action for infinitesimallystable maps used in [1, Section 1.6]. Additionally, local homeomorphisms arising from in-tegrating the above vector fields can be constrained tothose preserving a quadratic form (euclidean metric, orthe absolute quadric in the projective version) or aninvariant bilinear form (such as the symplectic form),then GL (3) × GL (4) can be replaced by the product ofthe corresponding affine or euclidean groups. Corollary 3
The tangent space to the double conju-gacy class
KPH − of any projection matrix P is the (3 × -matrix X · P − d P · Y , where · is the ordinaryproduct of matrices, and ( X , Y ) is the pair of fields ( k, h ) ∈ g (cid:96) (3) × g (cid:96) (4) corresponding to vector fields on ( K , H ) ∈ GL (3) × GL (4) . In our case X = ad ( K ) = ∇ K and Y = ad ( H ) = ∇ H , with a slight abuse of notation. Actually, this dou-ble action is implicit in the original formulation of theKLT-algorithm [25]. From the topological viewpoint, spatial homographiesdefine an open dense subset of P representing the or-dered (4 ×
4) array up to scale of entries of A ∈ P GL (4).The Lie algebra of endomorphisms corresponding to ho-mographies can also be stratified by the rank but onlya small 6D submanifold of these homographies arisefrom a rigid motion. Thus, information from internaland external camera parameters can be recovered withthe double conjugacy classes ( K , H ) of upper triangularmatrices K and euclidean transforms H .In order to include degeneracies in the space of pairsof endomorphisms, ( k , h ) must display some kind of“infinitesimal stability”. It suffices to prove that the(3 × X · P − d P · Y is always regular, i.e. ithas 3 as maximal rank. In this way, it fills out the tan-gent space to the left-right A -orbit of any projectionmatrix P . This result is true for the regular orbit, butnot necessarily for degeneracies in the Lie algebras rep-resented by degenerate End ( R ) × End ( R ). However,the construction of complete endomorphisms with theexterior powers removes the indeterminacy in degenera-cies of the fundamental matrix.5.2 Extending double conjugacy actionsIndeterminacies in projection matrices appears whentheir corresponding fundamental or essential matricesare rank-deficient. In order to avoid them, the left-rightequivalence for matrices (representing endomorphismsor automorphisms of vector spaces) must consider de-generate cases. In fact, this approach can be considered as a particular case of the left-right or A -equivalencefor smooth map-germs f : R n → R p . More specifically,the A := L × R -action is defined by f (cid:55)→ k ◦ f ◦ h − for any pair of diffeomorphisms ( k, h ) ∈ Dif f ( R n ) × Dif f ( R p ) that preserve a point (the origin).The linearization of the topological A -action (pairsof diffeomorphisms) is an algebraic A -action that can bedescribed in terms of pairs of automorphisms (acting ongroups). From an infinitesimal viewpoint (Lie algebra g ), they can be seen as pairs of endomorphisms actingon the supporting vector space of g , to be completed ifthe rank is deficient. This is an immediate consequenceof the description of T e Dif f ( R n ) = GL ( n ; R ) for thesource space (similarly for the target space) of any map.Orbits are created by both actions as double conjugacyclasses, but their meaning is not exactly the same: A -action allows true deformations, whereas A -action onlyallows linear deformations, which can be reinterpretedas perspective transformations, e.g.5.3 Incidence varieties and multilinear constraintsIncidence conditions between linear subspaces are givenby relations, such as L ai ⊂ L bj or L ai ∩ L bj (cid:54) = ∅ where a = dim ( L a ) and b = dim ( L b ). All of them are repre-sented by linear equations that can display deficiencyrank conditions. In our case, they can also be expressedwith multilinear constraints posed by the projectionsof the geometric objects of the scene (points and lines,mainly). Fundamental or essential matrices, trifocal ten-sor, or more generally multilinear tensors provide themost common examples for structural constraints be-tween corresponding elements, such as points and lines.Multilinear constraints are not easy to manage dueto the ambiguity in correspondences between objectsand the need of efficient optimization procedures. In-deed, both problems are related since the ambiguityis solved using enough functionally independent con-straints, which require optimization if the set of equa-tions is not minimal. The simplest example is the eight-point linear algorithm that estimates the fundamentalmatrix[10]. This matrix is a point F in the 7-dimensionalvariety F of P . A forward approach based in sevenpoints leads to a highly non-linear procedure which isunstable and more difficult to solve.Optimization procedures in the space of multilineartensors are required when redundant noisy informationis available; from the algebraic viewpoing, noise can belinked to the smallest or near-zero eigenvalues. Regu-lar tensors do not include degenerations in boundarycomponents, because they are elements on an open set.Thus, to include degenerate tensors it is necessary toextend regular analysis which is performed in terms of omplete Endomorphisms in Computer Vision 19 complete objects (extending complete endomorphisms,for simplest tensors given by matrices). The rest ofthis Section remarks how eventual degeneracies can besolved by completing the information with appropriatecompactifications of incidence varieties. To ease theirinterpretation we adopt a geometric language insteadof the more formal language of [24]. The simplest incidence variety in a projective space P n fulfills p ∈ (cid:96) , where (cid:96) ∈ Grass ( P n ) is a line in P n , i.e.an element of the Grassmannian of projective lines. Theset of pairs ( p , (cid:96) ) ∈ P n × Grass ( P n ) with p ∈ (cid:96) formsthe total space E ( γ ,n ) of the canonical bundle γ ,n of Grass ( P n ). A direct consequence of this constructionis the following result: Proposition 7
With the above notation: – Epipolar constraints for corresponding points are el-ements of the total space E ( γ , ) . – The set of pencils λ F + λ F connecting two de-generate fundamental matrices F , F ∈ Sing ( F ) isthe constraint of E ( γ , ) to F ⊂ P . The topologicalclosure of this set of pencils is the -secant varietyto F , including cases where F and F coalesce sothat the secant becomes a tangent. The construction of the incidence variety can beextended to any kind of Grassmannians
Grass k ( P n ),and flag manifolds denoted by B ( r , . . . , r k ) whose ele-ments are finite collections of nested subspaces L r ⊂ L r + r ⊂ . . . L r + ... + r k = P n , where ( r , . . . , r k ) is apartition of n + 1. If k = 1, then Grass ( P n ) = P n ,and if k = 2, then B ( k + 1 , n − k ) = Grass k ( P n ) withpartition ( k + 1 , n − k ) In particular, the first non-trivial example of a com-plete flag manifold is associated to the partition (1 , , V ⊂ V ⊂ V with projectivization p ∈ (cid:96) ⊂ π . The stabilizer subgroup of a generic ele-ment in B (1 , ,
1) are the 3 × K (up to scale for projective flags) acting at leftfor the double conjugacy action. This formulation pro-vides a locally symmetric structure for the left actionto model changes in camera calibration. For instance,changes in focal length can be interpreted in terms ofuni-parameter subgroups of the orbit in the associatedflag manifold. The notation B ( r , . . . , r k ) for flag manifold is not stan-dard; very often F ( r , . . . , r k ) (F for flag in English) or D ( r , . . . , r k ) (D for drapeau in French) generate confusionwith the space of Fundamental Matrices and the set of dis-tributions (of vector fields, e.g.). Thus, we have chosen B . These manifolds are homogeneous spaces serving asthe base space of a canonical bundle that extends theproperties of the simplest case of the previous para-graph. Indeed, they are “classifying spaces” not only forincidence, but for tangency conditions too. In general,any (eventually degenerated) collections of points andlines in the projective space can be viewed as config-urations in an appropriate flag manifold B ( r , . . . , r k ).In this case, a locally symmetric structure in terms ofcellular decompositions must be recovered to deal withdegeneracies. A cellular decomposition of a N -dimensional variety X splits the variety in a disjoint union of k -dimensionalcells ( e ki , ∂e ki ) ∼ top ( B k , S k − ) for 1 ≤ k ≤ N . Here S k − is the ( k − k -dimensional ball B k . Hence, each k -dimensional stra-tum of a locally symmetric variety X is a “replication”of a basic k -dimensional cell by some elementary alge-braic operation (ordinary or curved reflections, e.g.). Itcan be represented by an oriented graph.For example, the cellular decomposition of P is thecomplementary of a complete flag p ∈ (cid:96) ⊂ π ⊂ P .They are isomorphic to affine spaces A = (cid:96) \ p (one-dimensional cells), A = π \ (cid:96) (two-dimensional cells),and A = P \ π (three-dimensional cells), correspond-ing to the fixation of elements at infinity. Furthermore,such cells are invariant by the action of the affine group.A similar reasoning can be outlined using the reductionto the euclidean group by fixing the absolute quadric,the absolute conic and the circular points [9].A less trivial example is the cellular decompositionof the Grassmannian Grass ( P ) of lines (cid:96) ⊂ P , whichis the relative localization of lines (cid:96) w.r.t. a completeflag p ∈ (cid:96) ⊂ π ⊂ P . They are isomorphic to affinespaces given by the complementary of consecutive Schu-bert cycles σ ( a , a ) or their dual representation ( β , β )with 2 ≥ β ≥ β ≥ Grass ( (cid:51) ) the dual description ( β , β )of Schubert cycles can be easily visualized in an orientedgraph with nodes located at the vertices of an “increas-ing stair”. Hence, the cellular decomposition containsone 4D cell, one 3D cell, two 2D cells, and one 1D cell,which are described as follows: – (0 ,
0) does not impose conditions about lines so itrepresents the whole Grassmannian; – (1 ,
0) imposes the constraint (cid:96) ∩ (cid:96) (cid:54) = ∅ , which is ahyperplane section of the Grassmannian; – (2 ,
0) is { (cid:96) ∈ Grass ( (cid:51) ) | (cid:96) ⊂ π } , which imposestwo conditions and it is isomorphic to the dual of P – (1 ,
1) is { (cid:96) ∈ Grass ( (cid:51) ) | p ∈ (cid:96) } , which imposestwo conditions and it is isomorphic to an ordinaryprojective plane P ; – (2 ,
1) = { (cid:96) ∈ Grass ( (cid:51) ) | p ∈ (cid:96) ⊂ π } , whichimposes three conditions and it is isomorphic to aprojective line P ; – (0 ,
0) is the point representing the fixed line (cid:96) ofthe flag.The same decomposition can be formulated in theeuclidean terms on the underlying vector space V ⊂ R n +1 by selecting nested subspaces linked to the stan-dard reference { e , . . . , e as an oriented fixed flag. Fur-thermore, this decomposition can be extended to any Grass ( P n ) adding the vertices linked to n − Grass ( P ) and Grass ( P )) thatrepresent the ambient space for planar and volumetriccollineations, respectively. The epipolar constraint ( p , p (cid:48) ) ∈ P × P can be ex-pressed in the projective framework as T pFp (cid:48) = 0,where F is the fundamental matrix. But it can also beinterpreted as the simplest incidence relations p ∈ (cid:96) (cid:48) oras p (cid:48) ∈ (cid:96) in each projective plane, which are mutuallydual.The set of pairs ( p , (cid:96) ) fulfilling the epipolar con-straint are isomorphic to the total space of the canoni-cal bundle γ on each projective plane P . From a globalviewpoint, the evolution in space and time of the epipo-lar constraint can be described on the tangent space τ P to the projective plane. It is well-known that ε P ⊕ τ P (cid:39) γ ⊕ γ ⊕ γ , (3)where ε P is the trivial vector bundle on the projec-tive plane P , ⊕ is the Whitney sum of vector bundles,and γ is the canonical line bundle on P . This justi-fies the twist of a fundamental matrix for the secantline (uniparametric pencil) connecting a pair of cameralocations. In the euclidean space the epipolar constraint for cor-responding points p , p (cid:48) ∈ E × E is represented by T pEp (cid:48) = 0, where E ∈ E is the essential matrix linkedto each pair of views.More specifically, if { e i } ≤ i ≤ represents the canon-ical basis for E , then the collection of subspaces rep-resented by e , e ∧ e and e i ∧ e ∧ e are a positiveoriented flag. This flag can be viewed as the startingflag to interpret essential matrices in terms of rotationsand translations. Proposition 8
The local description of a essential ma-trix as a product of SO (3) and S can be formulatedglobally as follows:1. The essential manifold E is the unit sphere bundle S τ SO (3) of the tangent bundle τ SO (3) .2. The evolution of the tangent bundle is τ ( SO (3) × S ) (cid:39) so (3) ⊕ τ S .Proof The former is proved in [23]. The proof of thelatter is based on the embedding i : S (cid:44) → R , suchthat there exists an isomorphism τ S ⊕ N S (cid:39) i ∗ τ R = ε S where N S (cid:39) ε S is the normal bundle to the ordi-nary embedding i of S and ε S = ε S ⊕ ε S ⊕ ε S is the 3-rank trivial bundle on S . Thus, τ S ⊕ ε S (cid:39) ε S However any copy of ε S cannot be simplified since τ S is not topologically trivial, i.e. it is not isomorphicto the Whitney sum of two copies of ε S . The quotientby the action of Z (corresponding to the lifting of theantipodal map to the canonical bundle) gives the de-scription of the tangent space to the projective space.This result links the information of fundamental matri-ces with the similar information for essential matrices. Despite the fact that most results presented in this pa-per are posed in a static framework, our main mo-tivation arise from the indeterminacies appearing atbootstrapping a mobile calibrated camera in a non-structured environment. There are many approaches inthe state of the art based on perspective models workingin structured or man-made scenes that take advantageof the support provided by perspective lines, vanish-ing points, horizon lines, e.g. However, the problem be-comes more challenging in low-structured environmentswhere a “weighted combination” of homographies and omplete Endomorphisms in Computer Vision 21 fundamental matrices provides a practical solution forbootstrapping.Generally, planar or structured scenes are better ex-plained by a homography, whereas non-planar or un-structured scenes are better explained by a fundamen-tal matrix. Usually, it is preferable to not assume aspecific geometric model but to compute both of themin parallel [20]. Later, the best model is selected using asimple heuristic that avoids the low-parallax cases andthe well-known twofold ambiguity solution arising whenall points in a planar scene are closer to the cameracenters [17]. In low-parallax cases both models are notwell-constrained and the solution yields an initial cor-rupted map that should be rejected. Indeed, the qualityof the tracking relies heavily in the bootstrapping of thesystem and, more specifically, in the choice of the mostsuitable geometric model.Our heuristic approach is developed using weightedPL-paths in the secant variety
Sec (1 , F ) to the funda-mental variety F that fills out the whole space P = P End ( V ). Each secant line (cid:96) ∈ Gr ( P ) cuts out F generically in three elements, which can coalesce in adouble tangency point plus an ordinary point. The tan-gent hyperplane at each point of F ∈ F provides thehomography H related to the fundamental matrix F .Overall, the main challenge consists of retrieving avalid fundamental matrix when the estimation degener-ates into a 1-rank matrix, instead of the expected 2-rankmatrix (when the camera points towards a planar scene,e.g.). The most common approaches are based on theintroduction of additional sensors, the manual specifica-tion of two keyframes in a structured scene from whichthe system must bootstrap [15], or the perturbation ofthe camera pose to find a close keyframe from which torecover. Each approach has its own drawbacks:1. Additional sensors or devices improve the robust-ness and safety of the systems (a requirement forautonomous navigation), but it does not provide ascientific solution to the problem. Moreover, it is notalways possible to modify the hardware of a system.2. Manual selection of the initial keyframe pair breaksthe autonomy of the system since it requires theuser interaction to bootstrap. In addition, not allscenes contain a structured region to compute thehomography.3. The perturbation of the camera pose degrades thecontinuity in the image and scene flows since thedirection of the perturbation is randomized. Also, itignores the recent history of the trajectory, leavingthe system in an inconsistent state. Proposition 6provides an infinitesimally stable structural resultto avoid this problem. Our approach retrieve the recent history expressedin terms of the kinematics of the trajectory as a liftedpath from the group G of transformations to its tangentbundle G × g , which adds the unit vector linked to thegradient to avoid the indeterminacy in the completionof P End ( V ). In presence of uncertainty due to rank de-ficiency, the local secant cone to the regular part of thefundamental or the essential variety can be computed.Then, the shortest chord that minimizes the angle w.r.t.the precedent trajectory is selected to avoid abrupt dis-continuities which make more difficult the control.Actually, this approach is independent of the dimen-sion of the underlying vector space V , and thus it couldbe extended to support dynamic scenes with additionalstructural homogeneous constraints. It is also indepen-dent of the subgroup so it can also be adapted to otherclassical groups, such as SL (2) or SL (3), which leaveinvariant the area or volume elements, or even the sym-plectic groups leaving invariant Hamilton-Jacobi mo-tion equations [21]. Hence, motion analysis in dynamicenvironments can be performed theoretically using thesame approach. Degeneracies in fundamental and essential matrix are acommon issue for hand-held cameras traveling arounda non-controlled environment. These singular matri-ces can be incorporated to the analysis using resultsfrom Classical Algebraic Geometry. This paper intro-duces the concept of complete endomorphisms to man-age degeneracies, providing a geometric reinterpreta-tion in terms of secant varieties. Instead of looking atconfigurations of ( k + 1)-tuples of corresponding points,our alternative approach focuses on the projective ge-ometry of ambient spaces where tensors and projectionmaps live. The graph of the adjoint map for End ( V ) = T Aut ( V ) viewed as the gradient field for matrix spaceprovides the first example of completion. This construc-tion is applied to the fundamental F and the essential E varieties by adding limits of tangent directions ap-proached by secants in the ambient space P End ( V )Completions of regular transformations (automor-phisms in a Lie group G ) in terms of their tangentspaces (endomorphisms in the Lie algebra g := T e G ) arealso extended to include the degenerate cases for projec-tion matrices. These completions are always managed interms of rank stratifications of spaces of matrices. Ex-terior algebra of the underlying vector spaces and itsprojectivization provides a framework to manage thisrank stratification. These stratifications can be inter-preted geometrically in terms of secant subspaces andtheir adjacent tangent subspaces of any dimension. The simultaneous completion of the transformations w.r.t.left-right action A on the source and target spaces orcontact action K on the graph of the projection map P → P allows a more robust feedback between imageinterpretation and scene reconstruction.The constructions of this paper admit extension con-cerning several topics, such as analytic presentation oflimits of tangent spaces (by using appropriate compact-ifications, e.g.), intrinsic localization of degeneracy lociin terms of cellular decompositions (inverse image ofthe secant map, e.g.), preservation of locally symmetricstructure of complete spaces (involving the adjoint rep-resentations), intrinsic formulation of image and sceneflows (for mobile cameras, e.g.), or a relation betweenthe motion and structure tensors (in the moment-mapframework, e.g.). The results can also be adapted to thegeometry of different kinds of infinitesimal transforma-tions involving arbitrary deformations of the geometricmodels of views.These developments have a direct application forbootstrapping and tracking the transformations of acamera pose in video sequences recorded in non-restrictedenvironments with arbitrary movements. Therefore, farfrom being just a theoretical curiosity, the infinitesimalcompletion of regular transformations provides a natu-ral and continuous framework for a unified treatment ofkinematics. In particular, degenerate cases can be man-aged in motion prediction to increase the robustness ofVisual Odometry algorithms. References
1. Arnold, V., Varchenko, A., Gusein-Zade, S.: Singularitiesof Differentiable Maps: Volume I: The Classification ofCritical Points Caustics and Wave Fronts. Monographsin Mathematics. Birkh¨auser Boston (1985)2. Bahturin, Y.A., Shestakov, I.P., Zaicev, M.V.: Gradingson simple jordan and lie algebras. Journal of Algebra (2), 849–868 (2005)3. Dubbelman, G., Dorst, L., Pijls, H.: Manifold statisticsfor essential matrices. In: Computer Vision ECCV 2012,pp. 531–544. Springer (2012)4. Faugeras, O.: Three-dimensional computer vision: a geo-metric viewpoint. The MIT press (1993)5. Fløystad, G., Kileel, J.D., Ottaviani, G.: The chow formof the essential variety in computer vision. Journal ofSymbolic Computation , 97–119 (2018)6. Fulton, W.: Intersection theory, vol. 2. Springer Science& Business Media (2013)7. Griffiths, P., Harris, J.: Principles of algebraic geometry.John Wiley & Sons (2014)8. Guillemin, V., Sternberg, S.: Symplectic techniques inphysics. Cambridge university press (1990)9. Hartley, R., Zisserman, A.: Multiple view geometry incomputer vision. Cambridge university press (2003)10. Hartley, R.I.: In defense of the eight-point algorithm.IEEE Transactions on Pattern Analysis and Machine In-telligence (6), 580–593 (1997) 11. Kileel, J.D.: Algebraic geometry for computer vision.Ph.D. thesis, University of California, Berkeley (2017)12. Kim, J.S., Kanade, T.: Degeneracy of the linearseventeen-point algorithm for generalized essential ma-trix. Journal of Mathematical Imaging and Vision (1),40–48 (2010)13. Kirillov, A.A.: Lectures on the orbit method. In: Grad-uate Studies in Mathematics, vol. 64. American Mathe-matical Society (2004)14. Kirwan, F.C.: Cohomology of quotients in symplectic andalgebraic geometry, vol. 31. Princeton University Press(1984)15. Klein, G., Murray, D.: Parallel tracking and mappingfor small ar workspaces. In: 6th IEEE and ACM Inter-national Symposium on Mixed and Augmented Reality,2007. ISMAR 2007., pp. 225–234. IEEE (2007)16. Libermann, P., Marle, C.M.: Symplectic geometry andanalytical mechanics, vol. 35. Springer Science & Busi-ness Media (2012)17. Longuet-Higgins, H.: The reconstruction of a plane sur-face from two perspective projections. In: Proc. RoyalSoc. London Ser. B, Biol, Sci., vol. 227, pp. 399–410(1986)18. Ma, Y., Koˇseck´a, J., Sastry, S.: Optimization criteria andgeometric algorithms for motion and structure estima-tion. International Journal of Computer Vision (3),219–249 (2001)19. Miller, M.I., Younes, L.: Group actions, homeomor-phisms, and matching: A general framework. Interna-tional Journal of Computer Vision (1-2), 61–84 (2001)20. Mur-Artal, R., Montiel, J.M.M., Tardos, J.D.: Orb-slam:a versatile and accurate monocular slam system. IEEETransactions on Robotics (5), 1147–1163 (2015)21. Osher, S., Sethian, J.A.: Fronts propagating withcurvature-dependent speed: algorithms based onhamilton-jacobi formulations. Journal of computationalphysics (1), 12–49 (1988)22. Shafarevich, I.R.: Basic Algebraic Geometry 1, vol. 3.Berlin: Springer-Verlag (2013)23. Subbarao, R., Genc, Y., Meer, P.: Robust unambiguousparametrization of the essential manifold. In: IEEE Con-ference on Computer Vision and Pattern Recognition,2008., pp. 1–8. IEEE (2008)24. Thorup, A., Kleiman, S.: Complete bilinear forms. In: Al-gebraic Geometry Sundance 1986, pp. 253–320. Springer(1988)25. Tomasi, C., Kanade, T.: Shape and motion from imagestreams under orthography: a factorization method. In-ternational Journal of Computer Vision (2), 137–154(1992)26. Tron, R., Daniilidis, K.: The space of essential matricesas a riemannian quotient manifold. SIAM Journal onImaging Sciences (3), 1416–1445 (2017)27. Ye, K., Lim, L.H.: Schubert varieties and distances be-tween subspaces of different dimensions. SIAM Journalon Matrix Analysis and Applications (3), 1176–1197(2016)28. Zak, F.L.: Severi varieties. Sbornik: Mathematics54