A. Aydin Alatan

Image sequence analysis for emerging interactive multimedia services-the European COST 211 framework

Flexibility and efficiency of coding, content extraction, and content-based search are key research topics in the field of interactive multimedia. Ongoing ISO MPEG-4 and MPEG-7 activities are targeting standardization to facilitate such services. European COST Telecommunications activities provide a framework for research collaboration. At present a significant effort of the COST 211/sup ter/ group activities is dedicated toward image and video sequence analysis and segmentation-an important technological aspect for the success of emerging object-based MPEG-4 and MPEG-7 multimedia applications. The current work of COST 211 is centered around the test model, called the analysis model (AM). The essential feature of the AM is its ability to fuse information from different sources to achieve a high-quality object segmentation. The current information sources are the intermediate results from frame-based (still) color segmentation, motion vector based segmentation, and change-detection-based segmentation. Motion vectors, which form the basis for the motion vector based intermediate segmentation, are estimated from consecutive frames. A recursive shortest spanning tree (RSST) algorithm is used to obtain intermediate color and motion vector based segmentation results. A rule-based region processor fuses the intermediate results; a postprocessor further refines the final segmentation output. The results of the current AM are satisfactory.

Estimation of depth fields suitable for video compression based on 3-D structure and motion of objects

Intensity prediction along motion trajectories removes temporal redundancy considerably in video compression algorithms. In three-dimensional (3-D) object-based video coding, both 3-D motion and depth values are required for temporal prediction. The required 3-D motion parameters for each object are found by the correspondence-based E-matrix method. The estimation of the correspondences-two-dimensional (2-D) motion field-between the frames and segmentation of the scene into objects are achieved simultaneously by minimizing a Gibbs energy. The depth field is estimated by jointly minimizing a defined distortion and bit-rate criterion using the 3-D motion parameters. The resulting depth field is efficient in the rate-distortion sense. Bit-rate values corresponding to the lossless encoding of the resultant depth fields are obtained using predictive coding; prediction errors are encoded by a Lempel-Ziv algorithm. The results are satisfactory for real-life video scenes.

Object based 3-D motion and structure estimation

Motion analysis is the most crucial part of object-based coding. Motion in a 3-D environment can be analyzed better by using a 3-D motion model compared to its 2-D counterpart and hence may improve coding efficiency. Gibbs formulated joint segmentation and estimation of 2-D motion not only improves the performance, but also generates robust point correspondences which are necessary for linear 3-D motion estimation algorithms. Estimated 3-D motion parameters are used to find the structure of the previously segmented objects by minimizing another Gibbs energy. Such an approach achieves error immunity compared to linear algorithms. Experimental results are promising and hence the proposed motion and structure analysis method is a candidate to be used in object-based (or even knowledge-based) video coding schemes.

3-D motion estimation of rigid objects for video coding applications using an improved iterative version of the E-matrix method

As an alternative to current two-dimensional (2-D) motion models, a robust three-dimensional (3-D) motion estimation method is proposed to be utilized in object-based video coding applications. Since the popular E-matrix method is well known for its susceptibility to input errors, a performance indicator, which tests the validity of the estimated 3-D motion parameters both explicitly and implicitly, is defined. This indicator is utilized within the RANSAC method to obtain a robust set of 2-D motion correspondences which leads to better 3-D motion parameters for each object. The experimental results support the superiority of the proposed method over direct application of the E-matrix method.

Joint estimation and optimum encoding of depth field for 3-D object-based video coding

3-D motion models can be used to remove temporal redundancy between image frames. For efficient encoding using 3-D motion information, apart from the 3-D motion parameters, a dense depth field must also be encoded to achieve 2-D motion compensation on the image plane. Inspired by rate-distortion theory, a novel method is proposed to optimally encode the dense depth fields of the moving objects in the scene. Using two intensity frames and 3-D motion parameters as inputs, an encoded depth field can be obtained by jointly minimizing a distortion criteria and a bit-rate measure. Since the method gives directly an encoded field as an output, it does not require an estimate of the field to be encoded. By efficiently encoding the depth field during the experiments, it is shown that the 3-D motion models can be used in object based video compression algorithms.

Three-dimensional motion and dense-structure estimation using convex projections

We propose a novel method for estimating the 3D motion and dense structure of an object form its two 2D images. The proposed method is an iterative algorithm based on the theory of projections onto convex sets (POCS) that involves successive projections onto closed convex constraint sets. We seek a solution for the 3D motion and structure information that satisfies the following constraints: (i) rigid motion--the 3D motion parameters are the same for each point on the object. (ii) Smoothness of the structure--depth values of the neighboring points on the object vary smoothly. (iii) Temporal correspondence--the intensities in the given 2D images match under the 3D motion and structure parameters. We mathematically derive the projection operators onto these sets and discuss the convergence properties of successive projections. Experimental results show that the proposed method significantly improves the initial motion and structure estimates.

Gibbs Model Based 3D Motion and Structure Estimation for Object-Based Video Coding Applications

Motion analysis is essential for any video coding scheme. A moving object in a 3D environment can be analyzed better by a 3D motion model instead of 2D models, and better modeling might lead to improved coding efficiency. Gibbs formulated joint segmentation and estimation of 2D motion not only improves the performance of each stage, but also generates robust point correspondences which are necessary for rigid 3D motion estimation algorithms. Estimated rigid 3D motion parameters of a segmented object are used to find the 3D structure of those objects by minimizing another Gibbs energy. Such an approach achieves error immunity compared to linear algorithms. A more general (non-rigid) motion model can also be proposed using Gibbs formulation which permits local elastic interactions in contrast to ultimately tight rigidity between object points. Experimental results are promising for both rigid and non-rigid 3D motion models and put these models forward as strong candidates to be used in object-based coding algorithms.

Gibbs random field model based 3D motion estimation from video sequences

In contrast to previous global 3D motion concept, a Gibbs random field based method, which models local interactions between motion parameters defined at each point on the object, is proposed. An energy function which gives the joint probability distribution of motion vectors, is constructed. The energy function is minimized in order to find the most likely motion vector set. Some convergence problems, due to ill-posedness of the problem, are overcome by using the concept of hierarchical rigidity. In hierarchical rigidity, the objects are assumed to be almost rigid in the coarsest level and this rigidness is weakened at each level until the finest level is reached. The propagation of motion information between levels, is encouraged. At the finest level, each point have a motion vector associated with it and the interaction between these vectors are described by the energy function. The minimization of the energy function is achieved by using hierarchical rigidity, without trapping into a local minimum. The results are promising.