Enric Meinhardt-Llopis

TV-L1 Optical Flow Estimation

This article describes an implementation of the optical flow estimation method introduced by Zach, Pock and Bischof in 2007. This method is based on the minimization of a functional containing a data term using the L 1 norm and a regularization term using the total variation of the flow. The main feature of this formulation is that it allows discontinuities in the flow field, while being more robust to noise than the classical approach by Horn and Schunck. The algorithm is an efficient numerical scheme, which solves a relaxed version of the problem by alternate minimization. Source Code A C implementation of this algorithm is provided. The source code and an online demo are accessible at the web page of this article 1 .

Horn-Schunck Optical Flow with a Multi-Scale Strategy

The seminal work of Horn and Schunck is the first variational method for optical flow estimation. It introduced a novel framework where the optical flow is computed as the solution of a minimization problem. From the assumption that pixel intensities do not change over time, the optical flow constraint equation is derived. This equation relates the optical flow with the derivatives of the image. There are infinitely many vector fields that satisfy the optical flow constraint, thus the problem is ill-posed. To overcome this problem, Horn and Schunck introduced an additional regularity condition that restricts the possible solutions. Their method minimizes both the optical flow constraint and the magnitude of the variations of the flow field, producing smooth vector fields. One of the limitations of this method is that, typically, it can only estimate small motions. In the presence of large displacements, this method fails when the gradient of the image is not smooth enough. In this work, we describe an implementation of the original Horn and Schunck method and also introduce a multi-scale strategy in order to deal with larger displacements. For this multi-scale strategy, we create a pyramidal structure of downsampled images and change the optical flow constraint equation with a nonlinear formulation. In order to tackle this nonlinear formula, we linearize it and solve the method iteratively in each scale. In this sense, there are two common approaches: one that computes the motion increment in the iterations; or the one we follow, that computes the full flow during the iterations. The solutions are incrementally refined over the scales. This pyramidal structure is a standard tool in many optical flow methods.

Integral Images for Block Matching

The integral image representation is a remarkable idea that permits to evaluate the sum of image values over rectangular regions of the image with four operations, regardless of the size of the region. It was first proposed under the name of summed area table in the computer graphics community by Crow’84, in order to efficiently filter texture maps. Itwas later popularized in the computer vision community by Viola & Jones’04 with its use in their real-time object detection framework. In this article we describe the integral image algorithm and study its application in the context of block matching. We investigate tradeoffs and the limits of the performance gain with respect to exhaustive block matching. Source Code The source code and the online demo are accessible from the IPOL web page of this article 1 .

On stereo-rectification of pushbroom images

Image stereo pairs obtained from pinhole cameras can be stereo-rectified, thus permitting to test and use the many standard stereo matching algorithms of the literature. Yet, it is well-known that pushbroom Earth observation satellites produce image pairs that are not stereo-rectifiable. Nevertheless, we show that by a new and adequate use of the satellite calibration data, one can perform a precise local stereo-rectification of large Earth images. Based on this we built a fully automatic 3D reconstruction chain for the new Pléiades Earth observation satellite. It produces 1/10 pixel accurate Earth image stereo pairs at a high resolution. Examples will be made available online to the computer vision community.

FALDOI: A New Minimization Strategy for Large Displacement Variational Optical Flow

We propose a large displacement optical flow method that introduces a new strategy to compute a good local minimum of any optical flow energy functional. The method requires a given set of discrete matches, which can be extremely sparse, and an energy functional which locally guides the interpolation from those matches. In particular, the matches are used to guide a structured coordinate descent of the energy functional around these keypoints. It results in a two-step minimization method at the finest scale which is very robust to the inevitable outliers of the sparse matcher and able to capture large displacements of small objects. Its benefits over other variational methods that also rely on a set of sparse matches are its robustness against very few matches, high levels of noise, and outliers. We validate our proposal using several optical flow variational models. The results consistently outperform the coarse-to-fine approaches and achieve good qualitative and quantitative performance on the standard optical flow benchmarks.

Implementation of the Centroid Method for the Correction of Turbulence

The centroid method for the correction of turbulence consists in computing the Karcher-Fr echet mean of the sequence of input images. The direction of deformation between a pair of images is determined by the optical ow. A distinguishing feature of the centroid method is that it can produce useful results from an arbitrarily small set of input images. Source Code The source code and a online demo are accessible at the IPOL web page of this article 1 .

Poisson Image Editing

The gradient of images can be directly edited to perform useful operations; this is called gradientbased image processing or Poisson editing. For example operations such as seamless cloning, contrast enhancement, texture flattening or seamless tiling can be performed in a very simple and efficient way by combining/modifying the image gradients. In the present work we will describe the Poisson image editing method, and review the contributions that have been made since it was proposed in 2003. In addition the integration problem will be discussed and analyzed, both from the theoretical and numerical points of view. Two different numerical implementations will be discussed, the first one uses discrete versions of differential operators to convert the problem into a sparse linear system of equations, while the second one is based on Fourier transform properties. Source Code The Octave/Matlab source code, the code documentation, and the online demo are accessible at the IPOL web page of this article1 and usage instruction are included in the README.txt file of the compressed archive.

Automatic sensor orientation refinement of Pléiades stereo images

Modern Earth observation satellites are calibrated in such a way that a point on the ground can be located with an error of just a few pixels in the image domain. For many applications this error can be ignored, but this is not the case for stereo reconstruction, that requires sub-pixel accuracy. In this article we propose a method to correct this error. The method works by estimating local corrections that compensate the error relative to a reference image. The proposed method does not rely on ground control points, but only on the relative consistency of the image contents. We validate our method with Pléiades and WorldView-1 images on a representative set of geographic sites.

Relative depth from monocularoptical flow

We present a method to compute the relative depth of moving objects in video sequences. The method relies on the fact that the boundary between two moving objects follows the movement of the object which is closest to the camera. Thus, the input of the method is a segmentation (to know the boundaries of objects) and an optical flow (to know the movement of the objects). The output of the method is a relative ordering of the neighboring segments. In fact, this output only provides a cue of the desired relative ordering, just like T-junctions typically provide a cue of the relative ordering of the objects around them. These cues can be used later as heuristics or as starting points for higher-level algorithms for image and video-processing.

Automatic 3D Reconstruction from Multi-date Satellite Images

We propose an algorithm for computing a 3D model from several satellite images of the same site. The method works even if the images were taken at different dates with important lighting and vegetation differences. We show that with a large number of input images the resulting 3D models can be as accurate as those obtained from a single same-date stereo pair. To deal with seasonal vegetation changes, we propose a strategy that accounts for the multi-modal nature of 3D models computed from multi-date images. Our method uses a local affine camera approximation and thus focuses on the 3D reconstruction of small areas. This is a common setup in urgent cartography for emergency management, for which abundant multi-date imagery can be immediately available to build a reference 3D model. A preliminary implementation of this method was used to win the IARPA Multi-View Stereo 3D Mapping Challenge 2016. Experiments on the challenge dataset are used to substantiate our claims.