Ting-Chuen Pong

Detecting moving objects

The detection of moving objects is important in many tasks. This paper examines moving object detection based primarily on optical flow. We conclude that in realistic situations, detection using visual information alone is quite difficult, particularly when the camera may also be moving. The availability of additional information about camera motion and/or scene structure greatly simplifies the problem. Two general classes of techniques are examined. The first is based upon the motion epipolar constraint—translational motion produces a flow field radially expanding from a “focus of expansion” (FOE). Epipolar methods depend on knowing at least partial information about camera translation and/or rotation. The second class of methods is based on comparison of observed optical flow with other information about depth, for example from stereo vision. Examples of several of these techniques are presented.

A Markov random field image segmentation model for color textured images

Abstract We propose a Markov random field (MRF) image segmentation model, which aims at combining color and texture features. The theoretical framework relies on Bayesian estimation via combinatorial optimization (simulated annealing). The segmentation is obtained by classifying the pixels into different pixel classes. These classes are represented by multi-variate Gaussian distributions. Thus, the only hypothesis about the nature of the features is that an additive Gaussian noise model is suitable to describe the feature distribution belonging to a given class. Here, we use the perceptually uniform CIE-L * u * v * color values as color features and a set of Gabor filters as texture features. Gaussian parameters are either computed using a training data set or estimated from the input image. We also propose a parameter estimation method using the EM algorithm. Experimental results are provided to illustrate the performance of our method on both synthetic and natural color images.

On clustering and retrieval of video shots through temporal slices analysis

Based on the analysis of temporal slices, we propose novel approaches for clustering and retrieval of video shots. Temporal slices are a set of two-dimensional (2-D) images extracted along the time dimension of an image volume. They encode rich set of visual patterns for similarity measure. In this paper, we first demonstrate that tensor histogram features extracted from temporal slices are suitable for motion retrieval. Subsequently, we integrate both tensor and color histograms for constructing a two-level hierarchical clustering structure. Each cluster in the top level contains shots with similar color while each cluster in bottom level consists of shots with similar motion. The constructed structure is then used for the cluster-based retrieval. The proposed approaches are found to be useful particularly for sports games, where motion and color are important visual cues when searching and browsing the desired video shots.

Motion analysis and segmentation through spatio-temporal slices processing

This paper presents new approaches in characterizing and segmenting the content of video. These approaches are developed based upon the pattern analysis of spatio-temporal slices. While traditional approaches to motion sequence analysis tend to formulate computational methodologies on two or three adjacent frames, spatio-temporal slices provide rich visual patterns along a larger temporal scale. We first describe a motion computation method based on a structure tensor formulation. This method encodes visual patterns of spatio-temporal slices in a tensor histogram, on one hand, characterizing the temporal changes of motion over time, on the other hand, describing the motion trajectories of different moving objects. By analyzing the tensor histogram of an image sequence, we can temporally segment the sequence into several motion coherent subunits, in addition, spatially segment the sequence into various motion layers. The temporal segmentation of image sequences expeditiously facilitates the motion annotation and content representation of a video, while the spatial decomposition of image sequences leads to a prominent way of reconstructing background panoramic images and computing foreground objects.

Motion-Based Video Representation for Scene Change Detection

In this paper, we present a new framework to automatically group similar shots into one scene, where a scene is generally referred to as a group of shots taken place in the same site. Two major components in this framework are based on the motion characterization and background segmentation. The former component leads to an effective video representation scheme by adaptively selecting and forming keyframes. The later is considered novel in that background reconstruction is incorporated into the detection of scene change. These two components, combined with the color histogram intersection, establish our basic concept on assessing the similarity of scenes.

Video partitioning by temporal slice coherency

We present a novel approach for video partitioning by detecting three essential types of camera breaks, namely cuts, wipes, and dissolves. The approach is based on the analysis of temporal slices which are extracted from the video by slicing through the sequence of video frames and collecting temporal signatures. Each of these slices contains both spatial and temporal information from which coherent regions are indicative of uninterrupted video partitions separated by camera breaks. Properties could further be extracted from the slice for both the detection and classification of camera breaks. For example, cut and wipes are detected by color-texture properties, while dissolves are detected by statistical characteristics. The approach has been tested by extensive experiments.

Detection of gradual transitions through temporal slice analysis

In this paper, we present approaches for detecting camera cuts, wipes and dissolves based on the analysis of spatio-temporal slices obtained from videos. These slices are composed of spatially and temporally coherent regions which can be perceived as shots. In the proposed methods, camera breaks are located by performing color-texture segmentation and statistical analysis on these video slices. In addition to detecting camera breaks, our methods can classify the detected breaks as camera cuts, wipes and dissolves in an efficient manner.

On clustering and retrieval of video shots

Clustering of video data is an important issue in video abstraction, browsing and retrieval. In this paper, we propose a two-level hierarchical clustering approach by aggregating shots with similar motion and color features. Motion features are computed directly from 2D tensor histograms, while color features are represented by 3D color histograms. Cluster validity analysis is further applied to automatically determine the number of clusters at each level. Video retrieval can then be done directly based on the result of clustering. The proposed approach is found to be useful particularly for sports games, where motion and color are important visual cues when searching and browsing the desired video shots. Since most games involve two teams, clsssification and retrieval of teams becomes an interesting topic. To achieve these goals, nevertheless, an initial as well as critical step is to isolate team players from background regions. Thus, we also introduce approach to segment foreground objects (players) prior to classification and retrieval.

Exploiting image indexing techniques in DCT domain

Abstract This paper is concerned with the indexing and retrieval of images based on features extracted directly from the JPEG discrete cosine transform (DCT) domain. We examine possible ways of manipulating DCT coefficients by standard image analysis approaches to describe image shape, texture, and color. Through the Mandala transformation, our approach groups a subset of DCT coefficients to form ten blocks. Each block represents a particular frequency content of the original image. Two blocks are used to model rough object shape; nine blocks to describe subband properties; and one block to compute color distribution. As a result, the amount of data used for processing and analysis is significantly reduced. This can lead to simple yet efficient ways of indexing and retrieval in a large-scale image database. Experimental results show that our proposed approach offers superior indexing speed without significantly sacrificing the retrieval accuracy.

Color image segmentation and parameter estimation in a markovian framework

Abstract An unsupervised color image segmentation algorithm is presented, using a Markov random field (MRF) pixel classification model. We propose a new method to estimate initial mean vectors effectively even if the histogram does not have clearly distinguishable peaks. The only parameter supplied by the user is the number of classes.