Ze-Nian Li

Unsupervised Discovery of Action Classes

In this paper we consider the problem of describing the action being performed by human figures in still images. We will attack this problem using an unsupervised learning approach, attempting to discover the set of action classes present in a large collection of training images. These action classes will then be used to label test images. Our approach uses the coarse shape of the human figures to match pairs of images. The distance between a pair of images is computed using a linear programming relaxation technique. This is a computationally expensive process, and we employ a fast pruning method to enable its use on a large collection of images. Spectral clustering is then performed using the resulting distances. We present clustering and image labeling results on a variety of datasets.

Illumination-invariant color object recognition via compressed chromaticity histograms of color-channel-normalized images

Several color object recognition methods that are based on image retrieval algorithms attempt to discount changes of illumination in order to increase performance when test image illumination conditions differ from those that obtained when the image database was created. Here we extend the seminal method of Swain and Ballard to discount changing illumination. The new method is based on the first stage of the simplest color indexing method, which uses angular invariants between color image and edge image channels. That method first normalizes image channels, and then effectively discards much of the remaining information. Here we adopt the color-normalization stage as an adequate color constancy step. Further, we replace 3D color histograms by 2D chromaticity histograms. Treating these as images, we implement the method in a compressed histogram-image domain using a combination of wavelet compression and Discrete Cosine Transform (DCT) to fully exploit the technique of low-pass filtering for efficiency. Results are very encouraging, with substantially better performance than other methods tested. The method is also fast, in that the indexing process is entirely carried out in the compressed domain and uses a feature vector of only 36 or 72 values.

MultiMediaMiner: a system prototype for multimedia data mining

Multimedia data mining is the mining of high-level multimedia information and knowledge from large multimedia databases. A multimedia data mining system prototype, MultiMediaMiner, has been designed and developed. It includes the construction of a multimedia data cube which facilitates multiple dimensional analysis of multimedia data, primarily based on visual content, and the mining of multiple kinds of knowledge, including summarization, comparison, classification, association, and clustering.

Review and Preview: Disocclusion by Inpainting for Image-Based Rendering

Image-based rendering takes as input multiple images of an object and generates photorealistic images from novel viewpoints. This approach avoids explicitly modeling scenes by replacing the modeling phase with an object reconstruction phase. Reconstruction is achieved in two possible ways: recovering 3-D point locations using multiview stereo techniques, or reasoning about consistency of each voxel in a discretized object volume space. The most challenging problem for image-based reconstruction is the presence of occlusions. Occlusions make reconstruction ambiguous for object parts not visible in any input image. These parts must be reconstructed in a visually acceptable way. This paper both reviews image inpainting and argues that inpainting can provide not only attractive reconstruction but also a framework for increasing the accuracy of depth recovery. Digital image inpainting refers to any methods that fill-in holes of arbitrary topology in images so that they seem to be a part of the original image. Available methods are broadly classified as structural inpainting or textural inpainting. Structural inpainting reconstructs using prior assumptions and boundary conditions, while textural inpainting considers only the available data from texture exemplars or other templates. Of particular particular interest is research on structural inpainting applied to 3-D models, emphasizing its effectiveness for disocclusion.

Matching by Linear Programming and Successive Convexification

We present a novel convex programming scheme to solve matching problems, focusing on the challenging problem of matching in a large search range and with cluttered background. Matching is formulated as metric labeling with L1 regularization terms, for which we propose a novel linear programming relaxation method and an efficient successive convexification implementation. The unique feature of the proposed relaxation scheme is that a much smaller set of basis labels is used to represent the original label space. This greatly reduces the size of the searching space. A successive convexification scheme solves the labeling problem in a coarse to fine manner. Importantly, the original cost function is reconvexified at each stage, in the new focus region only, and the focus region is updated so as to refine the searching result. This makes the method well-suited for large label set matching. Experiments demonstrate successful applications of the proposed matching scheme in object detection, motion estimation, and tracking

Successive Convex Matching for Action Detection

We propose human action detection based on a successive convex matching scheme. Human actions are represented as sequences of postures and specific actions are detected in video by matching the time-coupled posture sequences to video frames. The template sequence to video registration is formulated as an optimal matching problem. Instead of directly solving the highly non-convex problem, our method convexifies the matching problem into linear programs and refines the matching result by successively shrinking the trust region. The proposed scheme represents the target point space with small sets of basis points and therefore allows efficient searching. This matching scheme is applied to robustly matching a sequence of coupled binary templates simultaneously in a video sequence with cluttered backgrounds.

Illumination–invariant image retrieval and video segmentation

Abstract Images or videos may be imaged under different illuminants than models in an image or video proxy database. Changing illumination color in particular may confound recognition algorithms based on color histograms or video segmentation routines based on these. Here we show that a very simple method of discounting illumination changes is adequate for both image retrieval and video segmentation tasks. We develop a feature vector of only 36 values that can also be used for both these objectives as well as for retrieval of video proxy images from a database. The new image metric is based on a color-channel-normalization step, followed by reduction of dimensionality by going to a chromaticity space. Treating chromaticity histograms as images , we perform an effective low-pass filtering of the histogram by first reducing its resolution via a wavelet-based compression and then by a DCT transformation followed by zonal coding. We show that the color constancy step – color band normalization – can be carried out in the compressed domain for images that are stored in compressed form, and that only a small amount of image information need be decompressed in order to calculate the new metric. The new method performs better than previous methods tested for image or texture recognition and operates entirely in the compressed domain, on feature vectors. Apart from achieving illumination invariance for video segmentation, so that, e.g.an actor stepping out of a shadow does not trigger the declaration of a false cut, the metric reduces all videos to a uniform scale. Thus thresholds can be developed for a training set of videos and applied to any new video, including streaming video, for segmentation as a one-pass operation.

Illumination Invariance and Object Model in Content-Based Image and Video Retrieval

With huge amounts of multimedia information connected to the global information network (Internet), efficient and effective image retrieval from large image and video repositories has become an imminent research issue. This article presents our research in the C-BIRD (content-based image retrieval in digital-libraries) project. In addition to the use of common features such as color, texture, shape, and their conjuncts, and the combined content-based and description-based techniques, it is shown that (a) color-channel-normalization enables search by illumination invariance, and (b) feature localization and a three-step matching algorithm (color hypothesis, texture support, shape verification) facilitate search by object model in image and video databases.

Analysis of disparity gradient based cooperative stereo

This paper argues that the disparity gradient subsumes various constraints for stereo matching, and can thus be used as the basis of a unified cooperative stereo algorithm. Traditionally, selection of the neighborhood support function (NSF) in cooperative stereo was left as a heuristic exercise. We present an analysis and evaluation of three families of NSFs based on the disparity gradient. It is shown that an exponential decay function with a conveniently selectable parameter is well behaved in that it yields the least error, converges steadily, and produces correctly located weak-winners. The discovery of the well-behaved function facilitates the success of the disparity gradient based approach. It is suggested that this function will help a two-pass algorithm in resolving the dilemma of surface continuity and discontinuity/occlusion. In our experiments, the unified cooperative stereo-matching algorithm is tested on random-dot stereograms containing opaque and transparent surfaces. It is also shown to be applicable to both area matching and contour matching in real-world images.

A survey of motion-parallax-based 3-D reconstruction algorithms

The task of recovering three-dimensional (3-D) geometry from two-dimensional views of a scene is called 3-D reconstruction. It is an extremely active research area in computer vision. There is a large body of 3-D reconstruction algorithms available in the literature. These algorithms are often designed to provide different tradeoffs between speed, accuracy, and practicality. In addition, even the output of various algorithms can be quite different. For example, some algorithms only produce a sparse 3-D reconstruction while others are able to output a dense reconstruction. The selection of the appropriate 3-D reconstruction algorithm relies heavily on the intended application as well as the available resources. The goal of this paper is to review some of the commonly used motion-parallax-based 3-D reconstruction techniques and make clear the assumptions under which they are designed. To do so efficiently, we classify the reviewed reconstruction algorithms into two large categories depending on whether a prior calibration of the camera is required. Under each category, related algorithms are further grouped according to the common properties they share.