Harpreet S. Sawhney

Query by image and video content: the QBIC system

Research on ways to extend and improve query methods for image databases is widespread. We have developed the QBIC (Query by Image Content) system to explore content-based retrieval methods. QBIC allows queries on large image and video databases based on example images, user-constructed sketches and drawings, selected color and texture patterns, camera and object motion, and other graphical information. Two key properties of QBIC are (1) its use of image and video content-computable properties of color, texture, shape and motion of images, videos and their objects-in the queries, and (2) its graphical query language, in which queries are posed by drawing, selecting and other graphical means. This article describes the QBIC system and demonstrates its query capabilities. QBIC technology is part of several IBM products. >

Efficient color histogram indexing for quadratic form distance functions

An improved shipping container having novel locking features in the end panel and corner flaps. The novel locking features improved bulge resistance at the end panels from sideward bulge of the product. The improved container comprises a pair of corner flaps being hinged from the side panels and folded inwardly against an end panel with the two corner flaps and end panel on each side of the container having a quadruple lock. The lock is formed by providing a locking tab on each corner flap as well as a pair of locking tabs on the end panel with the end panel and corner flaps locking tabs being designed to be swung and locked in the opening formed by the aligned mating locking tab.In image retrieval based on color, the weighted distance between color histograms of two images, represented as a quadratic form, may be defined as a match measure. However, this distance measure i...

Layered representation of motion video using robust maximum-likelihood estimation of mixture models and MDL encoding

Representing and modeling the motion and spatial support of multiple objects and surfaces from motion video sequences is an important intermediate step towards dynamic image understanding. One such representation, called layered representation, has recently been proposed. Although a number of algorithms have been developed for computing these representations, there has not been a consolidated effort into developing a precise mathematical formulation of the problem. This paper presents one such formulation based on maximum likelihood estimation (MLE) of mixture models and the minimum description length (MDL) encoding principle. The three major issues in layered motion representation are: (i) how many motion models adequately describe image motion, (ii) what are the motion model parameters, and (iii) what is the spatial support layer for each motion model.<<ETX>>

Model-based 2D&3D dominant motion estimation for mosaicing and video representation

It is fairly common in video sequences that a mostly fixed background (scene) is imaged with or without objects. The dominant background changes in the image plane mostly due to camera operations and motion (zoom, pan, tilt, track etc.). We address the problem of computation of the dominant image transformation over time and demonstrate how this can be effectively used for efficient video representation through video mosaicing and image registration. We formulate the problem of dominant component estimation as that of model based robust estimation using M estimators with direct, multi resolution methods. In addition to 2D affine and plane projective models, that have been used in the past for describing image motion using direct methods, we also employ a true 3D model of motion and scene structure imaged with uncalibrated cameras. This model parameterizes the image motion as that due to a planar component and a parallax component. For rigid 3D scenes imaged under camera motion only, least squares (LS) methods with the plane and parallax parameterization are also presented. Furthermore, in the context of robust estimation, in contrast with previous approaches for similar problems, our algorithm employs an automatic computation of a scale parameter that is crucial in rejecting the non dominant components as outliers.<<ETX>>

Dynamic depth recovery from multiple synchronized video streams

This paper addresses the problem of extracting depth information of non-rigid dynamic 3D scenes from multiple synchronized video streams. Three main issues are discussed in this context. (i) temporally consistent depth estimation, (ii) sharp depth discontinuity estimation around object boundaries, and (iii) enforcement of the global visibility constraint. We present a framework in which the scene is modeled as a collection of 3D piecewise planar surface patches induced by color based image segmentation. This representation is continuously estimated using an incremental formulation in which the 3D geometric, motion, and global visibility constraints are enforced over space and time. The proposed algorithm optimizes a cost function that incorporates the spatial color consistency constraint and a smooth scene motion model.

Dynamic layer representation with applications to tracking

A dynamic layer representation is proposed for tracking moving objects. Previous work on layered representations has largely concentrated on two-/multi-frame batch formulations, and tracking research has not addressed the issue of joint estimation of object motion ownership and appearance. The paper extends the estimation of layers in a dynamic scene to incremental estimation formulation and demonstrates how this naturally solves the tracking problem. The three components of the dynamic layer representation, namely, layer motion, ownership, and appearance, are estimated simultaneously over time in a MAP framework. In order to enforce a global shape constraint and to maintain the layer segmentation over time, a parametric segmentation prior is proposed. The generalized EM algorithm is employed to compute the optimal solution. We show the results on real-time tracking of multiple moving or static objects in a cluttered scene imaged from a moving aerial video camera. The moving objects may do complex motions, and have complex interactions such as passing. By using both the appearance and the segmentation information, many difficult tracking tasks are reliably handled.

Efficient color histogram indexing

In image retrieval based on color, the weighted distance between color histograms of two images, represented as a quadratic form, may be defined as a match measure. However, this distance measure is computationally expensive (naively O(N/sup 2/) and at best O(N) in the number N of histogram bins) and it operates on high dimensional features (O(N)). We propose the use of low-dimensional, simple to compute distance measures between the color distributions, and show that these are lower bounds on the histogram distance measure. Results on color histogram matching in large image databases show that pre-filtering with the simpler distance measures leads to significantly less time complexity because the quadratic histogram distance is now computed on a smaller set of images. The low-dimensional distance measure can also be used for indexing into the database.<<ETX>>

Registration of video to geo-referenced imagery

The ability to locate scenes and objects visible in aerial video imagery with their corresponding locations and coordinates in a reference coordinate system is important in visually-guided navigation, surveillance and monitoring systems. However, a key technical problem of locating objects and scenes in a video with their geo-coordinates needs to be solved in order to ascertain the geo-location of objects seen from the camera platforms current location. We present the key algorithms for the problem of accurate mapping between camera coordinates and geo-coordinates, called geo-spatial registration. Current systems for geo-location use the position and attitude information for the moving platform in some fixed world coordinates to locate the video frames in the reference database. However, the accuracy achieved is only of the order of 100s of pixels. Our approach utilizes the imagery and terrain information contained in the geo-spatial database to precisely align dynamic videos with the reference imagery and thus achieves a much higher accuracy. Applications of geo-spatial registration include aerial mapping, target location and tracking and enhanced visualization such as the overlay of textual/graphical annotations of objects of interest in the current video using the stored annotations in the reference database.

Simplifying multiple motion and structure analysis using planar parallax and image warping

Presents a formulation for 3D motion and structure analysis using motion parallax defined with respect to an arbitrary plane in the environment. It is shown that if an image coordinate system is warped using plane projective transformation with respect to a reference view, the residual image motion is dependent only on the epipoles and has a simple relation to the 3D structure. Our computational scheme avoids point/line correspondence and is based on hierarchical estimation and image warping working directly with spatiotemporal image intensities. Results on real images demonstrate how this analysis can be used to simplify ego and object motion segmentation.<<ETX>>

Correlation-based estimation of ego-motion and structure from motion and stereo

This paper describes a correlation-based, iterative, multi-resolution algorithm which estimates both scene structure and the motion of the camera rig through an environment from the stream(s) of incoming images. Both single-camera rigs and multiple-camera rigs can be accommodated. The use of multiple synchronized cameras results in more rapid convergence of the iterative approach. The algorithm uses a global ego-motion constraint to refine estimates of inter-frame camera rotation and translation. It uses local window-based correlation to refine the current estimate of scene structure. All analysis is performed at multiple resolutions. In order to combine, in a straightforward way, the correlation surfaces from multiple viewpoints and from multiple pixels in a support region, each pixels correlation surface is modeled as a quadratic. This parameterization allows direct, explicit computation of incremental refinements for ego-motion and structure using linear algebra. Batches can be of arbitrary size, allowing a trade-off between accuracy and latency. Batches can also be daisy-chained for extended sequences. Results of the algorithm are shown on synthetic and real outdoor image sequences.