Quang-Tuan Luong

A robust technique for matching two uncalibrated images through the recovery of the unknown epipolar geometry

Abstract This paper proposes a robust approach to image matching by exploiting the only available geometric constraint, namely, the epipolar constraint. The images are uncalibrated, namely the motion between them and the camera parameters are not known. Thus, the images can be taken by different cameras or a single camera at different time instants. If we make an exhaustive search for the epipolar geometry, the complexity is prohibitively high. The idea underlying our approach is to use classical techniques (correlation and relaxation methods in our particular implementation) to find an initial set of matches, and then use a robust technique—the Least Median of Squares (LMedS)—to discard false matches in this set. The epipolar geometry can then be accurately estimated using a meaningful image criterion. More matches are eventually found, as in stereo matching, by using the recovered epipolar geometry. A large number of experiments have been carried out, and very good results have been obtained. Regarding the relaxation technique, we define a new measure of matching support, which allows a higher tolerance to deformation with respect to rigid transformations in the image plane and a smaller contribution for distant matches than for nearby ones. A new strategy for updating matches is developed, which only selects those matches having both high matching support and low matching ambiguity. The update strategy is different from the classical “winner-take-all”, which is easily stuck at a local minimum, and also from “loser-take-nothing”, which is usually very slow. The proposed algorithm has been widely tested and works remarkably well in a scene with many repetitive patterns.

Camera Self-Calibration: Theory and Experiments

The problem of finding the internal orientation of a camera (camera calibration) is extremely important for practical applications. In this paper a complete method for calibrating a camera is presented. In contrast with existing methods it does not require a calibration object with a known 3D shape. The new method requires only point matches from image sequences. It is shown, using experiments with noisy data, that it is possible to calibrate a camera just by pointing it at the environment, selecting points of interest and then tracking them in the image as the camera moves. It is not necessary to know the camera motion.

The Fundamental Matrix: Theory, Algorithms, and Stability Analysis

In this paper we analyze in some detail the geometry of a pair of cameras, i.e., a stereo rig. Contrarily to what has been done in the past and is still done currently, for example in stereo or motion analysis, we do not assume that the intrinsic parameters of the cameras are known (coordinates of the principal points, pixels aspect ratio and focal lengths). This is important for two reasons. First, it is more realistic in applications where these parameters may vary according to the task (active vision). Second, the general case considered here, captures all the relevant information that is necessary for establishing correspondences between two pairs of images. This information is fundamentally projective and is hidden in a confusing manner in the commonly used formalism of the Essential matrix introduced by Longuet-Higgins (1981). This paper clarifies the projective nature of the correspondence problem in stereo and shows that the epipolar geometry can be summarized in one 3×3 matrix of rank 2 which we propose to call the Fundamental matrix.After this theoretical analysis, we embark on the task of estimating the Fundamental matrix from point correspondences, a task which is of practical importance. We analyze theoretically, and compare experimentally using synthetic and real data, several methods of estimation. The problem of the stability of the estimation is studied from two complementary viewpoints. First we show that there is an interesting relationship between the Fundamental matrix and three-dimensional planes which induce homographies between the images and create unstabilities in the estimation procedures. Second, we point to a deep relation between the unstability of the estimation procedure and the presence in the scene of so-called critical surfaces which have been studied in the context of motion analysis. Finally we conclude by stressing the fact that we believe that the Fundamental matrix will play a crucial role in future applications of three-dimensional Computer Vision by greatly increasing its versatility, robustness and hence applicability to real difficult problems.

Robust recovery of the epipolar geometry for an uncalibrated stereo rig

This paper addresses the problem of accurately and automatically recovering the epipolar geometry from an uncalibrated stereo rig and its application to the image matching problem. A robust correlation based approach that eliminates outliers is developed to produce a reliable set of corresponding high curvature points. These points are used to estimate the so-called Fundamental Matrix which is closely related to the epipolar geometry of the uncalibrated stereo rig. We show that an accurate determination of this matrix is a central problem. Using a linear criterion in the estimation of this matrix is shown to yield erroneous results. Different parameterization and non-linear criteria are then developed to take into account the specific constraints of the Fundamental Matrix providing more accurate results. Various experimental results on real images illustrates the approach.

Motion of points and lines in the uncalibrated case

In the present paper we address the problem of computing structure and motion, given a set point and/or line correspondences, in a monocular image sequence, when the camera is not calibrated.Considering point correspondences first, we analyse how to parameterize the retinal correspondences, in function of the chosen geometry: Euclidean, affine or projective geometry. The simplest of these parameterizations is called the FQs-representation and is a composite projective representation. The main result is that considering N+1 views in such a monocular image sequence, the retinal correspondences are parameterized by 11 N−4 parameters in the general projective case. Moreover, 3 other parameters are required to work in the affine case and 5 additional parameters in the Euclidean case. These 8 parameters are “calibration” parameters and must be calculated considering at least 8 external informations or constraints. The method being constructive, all these representations are made explicit.Then, considering line correspondences, we show how the the same parameterizations can be used when we analyse the motion of lines, in the uncalibrated case. The case of three views is extensively studied and a geometrical interpretation is proposed, introducing the notion of trifocal geometry which generalizes the well known epipolar geometry. It is also discussed how to introduce line correspondences, in a framework based on point correspondences, using the same equations.Finally, considering the F Qs-representation, one implementation is proposed as a “motion module”, taking retinal correspondences as input, and providing and estimation of the 11 N−4 retinal motion parameters. As discussed in this paper, this module can also estimate the 3D depth of the points up to an affine and projective transformation, defined by the 8 parameters identified in the first section. Experimental results are provided.

Determining the fundamental matrix with planes: instability and new algorithms

The fundamental matrix is a key concept when working with uncalibrated images and multiple viewpoints. It contains all the available geometric information and enables recovery of the epipolar geometry from uncalibrated perspective views. The problem of its determination from points which lie in several planes is discussed. In that case, there is an homography between coordinates of points in the two images. The use of different criteria to compute the homography is investigated. A very simple and important relation between the homography matrices obtained from the observation of planar surfaces and the fundamental matrix is then established. Using simulations and real images to validate this analysis, it is shown that as a first consequence of this relation, the general methods to compute the fundamental matrix are unstable when the points lie close to planes. New algorithms are proposed to exploit this situation through the use of the previous relation. Their performance is compared to the performance of the general algorithm using a large number of noisy synthetic data and real images.<<ETX>>

Self-calibration of a camera using multiple images

The problem of calibrating cameras is extremely important in computer vision. Existing work is based on the use of a calibration pattern whose 3D model is known a priori. The authors present a complete method for calibrating a camera, which requires only point matches from image sequences. The authors show, using experiments with noisy data, that it is possible to calibrate a camera just by pointing it at the environment, selecting points of interests, and tracking them in the image while moving the camera with an unknown motion. The camera calibration is computed in two steps. In the first step the epipolar transformation is found via the estimation of the fundamental matrix. The second step of the computation uses the so-called Kruppa equations, which link the epipolar transformation to the intrinsic parameters. These equations are integrated in an iterative filtering scheme.<<ETX>>

Self-Calibration of a Stereo Rig from Unknown Camera Motions and Point Correspondences

The problem of calibrating a stereo rig is extremely important for practical applications. Existing work is based on the use of a calibration pattern whose 3D model is a priori known. We show theoretically and with experiments on real images, how it is possible to completely calibrate a stereo rig, that is to determine each camera’s intrinsic parameters and the relative displacement between the two or three cameras, using only point matches obtained during unknown motions, without any a priori knowledge of the scenes.

Motion of an uncalibrated stereo rig: self-calibration and metric reconstruction

We address in this paper the problem of self-calibration and metric reconstruction (up to a scale) from one unknown motion of an uncalibrated stereo rig. The epipolar constraint is first formulated for two uncalibrated images. The problem then becomes one of estimating unknowns such that the discrepancy from the epipolar constraint, in terms of sum of squared distances between points and their corresponding epipolar lines, is minimized. Redundancy of the information contained in a sequence of stereo images makes this method more robust than using a sequence of monocular images. Real data have been used to test the proposed method, and the results obtained are quite good.

Computing motion and structure in image sequences without calibration

This paper proposes an algebraic method to generalize the usual equations of structure from motion, when calibration is not available. Contrary to previous approaches, the construction is made without any geometry and does not require a deep understanding of complex abstract objects. The construction being well formalized, an effective algorithm is easily derived, and experimental results are shown.