John L. Barron

Performance of optical flow techniques

While different optical flow techniques continue to appear, there has been a lack of quantitative evaluation of existing methods. For a common set of real and synthetic image sequences, we report the results of a number of regularly cited optical flow techniques, including instances of differential, matching, energy-based, and phase-based methods. Our comparisons are primarily empirical, and concentrate on the accuracy, reliability, and density of the velocity measurements; they show that performance can differ significantly among the techniques we implemented.

The computation of optical flow

Two-dimensional image motion is the projection of the three-dimensional motion of objects, relative to a visual sensor, onto its image plane. Sequences of time-orderedimages allow the estimation of projected two-dimensional image motion as either instantaneous image velocities or discrete image displacements. These are usually called the optical flow field or the image velocity field. Provided that optical flow is a reliable approximation to two-dimensional image motion, it may then be used to recover the three-dimensional motion of the visual sensor (to within a scale factor) and the three-dimensional surface structure (shape or relative depth) through assumptions concerning the structure of the optical flow field, the three-dimensional environment, and the motion of the sensor. Optical flow may also be used to perform motion detection, object segmentation, time-to-collision and focus of expansion calculations, motion compensated encoding, and stereo disparity measurement. We investigate the computation of optical flow in this survey: widely known methods for estimating optical flow are classified and examined by scrutinizing the hypothesis and assumptions they use. The survey concludes with a discussion of current research issues.

Range Flow Estimation

Abstract We discuss the computation of the instantaneous 3D displacement vector fields of deformable surfaces from sequences of range data. We give a novel version of the basic motion constraint equation that can be evaluated directly on the sensor grid. The various forms of the aperture problem encountered are investigated and the derived constraint solutions are solved in a total least squares (TLS) framework. We propose a regularization scheme to compute dense full flow fields from the sparse TLS solutions. The performance of the algorithm is analyzed quantitatively for both synthetic and real data. Finally we apply the method to compute the 3D motion field of living plant leaves.

Quantitative color optical flow

We perform a qualitative and quantitative analysis of various multi-frame color optical flow methods for synthetic and real panning and zooming image sequences. We show that optical flow accuracy improvement can be slightly improved if color images are available instead of gray value or saturation images. We show the usefulness of a directional regularization constraint for computing optical flow when the camera motion is known to be panning or zooming.

Performance characterization in computer vision: A guide to best practices

It is frequently remarked that designers of computer vision algorithms and systems cannot reliably predict how algorithms will respond to new problems. A variety of reasons have been given for this situation and a variety of remedies prescribed in literature. Most of these involve, in some way, paying greater attention to the domain of the problem and to performing detailed empirical analysis. The goal of this paper is to review what we see as current best practices in these areas and also suggest refinements that may benefit the field of computer vision. A distinction is made between the historical emphasis on algorithmic novelty and the increasing importance of validation on particular data sets and problems.

The feasibility of motion and structure from noisy time-varying image velocity information

This research addresses the problem of noise sensitivity inherent in motion and structure algorithms. The motion and structure paradigm is a two-step process. First, we measure image velocities and, perhaps, their spatial and temporal derivatives, are obtained from time-varying image intensity data and second, we use these data to compute the motion of a moving monocular observer in a stationary environment under perspective projection, relative to a single 3-D planar surface. The first contribution of this article is an algorithm that uses time-varying image velocity information to compute the observers translation and rotation and the normalized surface gradient of the 3-D planar surface. The use of time-varying image velocity information is an important tool in obtaining a more robust motion and structure calculation. The second contribution of this article is an extensive error analysis of the motion and structure problem. Any motion and structure algorithm that uses image velocity information as its input should exhibit error sensitivity behavior compatible with the results reported here. We perform an average and worst case error analysis for four types of image velocity information: full and normal image velocities and full and normal sets of image velocity and its derivatives. (These derivatives are simply the coefficients of a truncated Taylor series expansion about some point in space and time.) The main issues we address here are: just how sensitive is a motion and structure computation in the presence of noisy input, or alternately, how accurate must our image velocity information be, how much and what type of input data is needed, and under what circumstances is motion and structure feasible? That is, when can we be sure that a motion and structure computation will produce usable results? We base our answers on a numerical error analysis we conduct for a large number of motions.

A High Resolution Dynamic Heart Model Based on Averaged MRI Data

We are in the process of constructing a high resolution, high signal to noise ratio (SNR) dynamic MRI dataset for the human heart using methodology similar to that employed to construct a low-noise standard brain at the Montreal Neurological Institute. Several high resolution, low SNR magnetic resonance images of 20 phases over the cardiac cycle were acquired from a single subject. Images from identical phases and temporally adjacent phases were registered, and the image intensities were averaged together to generate a high resolution, high SNR dynamic magnetic resonance image volume of the human heart. Although this work is still preliminary, and the results still demonstrate residual artifacts due to motion an sub-optimal alignment of interleaved image slices, our model has a SNR that is improved by a factor of 2.7 over a single volume, spatial resolution of 1.5 mm3, and a temporal resolution of 60 ms.

The frequency structure of one-dimensional occluding image signals

We present a theoretical investigation of the frequency structure of 1D occluding image signals. We show that image signal occlusion contains relevant information which is most easily extractable from its representation in the frequency domain. For instance, the occluding and occluded signal velocities may be identified as such and translucency phenomena may be understood in the terms of this theoretical investigation. In addition, it is found that the structure of occluding 1D signals is invariant under constant and linear models of signal velocity. This theoretical framework can be used to describe the exact frequency structure of non-Fourier motion and bridges the gap between such visual phenomena and their understanding in the frequency domain.

Dense range flow from depth and intensity data

The combined use of intensity and depth information greatly helps in the estimation of the local 3D movements (range flow) of moving surfaces. We demonstrate how the two can be combined in both: a local total least squares algorithm, and an iterative global variational technique. While the former assumes locally constant flow, the latter relies on a smoothly varying flow field. The improvement achieved through incorporating intensity is illustrated qualitatively and quantitatively on synthetic and real test data.

Regularised Range Flow

Extending a differential total least squares method for range flow estimation we present an iterative regularisation approach to compute dense range flow fields. We demonstrate how this algorithm can be used to detect motion discontinuities. This can be used to segment the data into independently moving regions. The different types of aperture problem encountered are discussed. Our regularisation scheme then takes the various types of flow vectors and combines them into a smooth flow field within the previously segmented regions. A quantitative performance analysis is presented on both synthetic and real data. The proposed algorithm is also applied to range data from castor oil plants obtained with the Biris laser range sensor to study the 3-D motion of plant leaves.