Nilesh V. Patel

A fast block-based motion compensation video frame interpolation

Skipped video frame reconstruction has been investigated under the limit of network bandwidth and media storage capability. One scenario is video conferencing, where frames are skipped to meet the transmission bandwidth which ends up with a jerky sequence. In this paper, the authors propose a new motion compensated frame rate up-conversion algorithm. Unlike the conventional approaches, the proposed technique investigates the quality and complexity tradeoff of a modified block based motion-compensated interpolation scheme. The time complexities are measured with respect to the quality of reconstructed video sequences by changing various thresholds. Empirical results demonstrate superiority of the proposed improvement.

Video frame rate up conversion under inconsistent camera motion

In this paper, we address the problem of video frame rate up-conversion (FRC) in the compressed domain. FRC is often recognized as video temporal interpolation. This problem is very challenging when targeted for video sequences with inconsistent camera and object motion, such as sports videos. A novel compressed domain motion compensation scheme is presented and applied in this paper, aiming at up-sampling frame rates in sports videos. MPEG-2 encoded motion vectors (MVs) are utilized as inputs in the proposed algorithm. The decoded MVs undergo a cumulative spatiotemporal interpolation. An iterative rejection scheme based on the dense motion vector field (MVF) and the generalized affine motion model is exploited to detect global camera motion. Subsequently, the foreground object separation is performed by additionally examining the temporal consistency of the output of iterative rejections. This consistency check process helps coalesce the resulting foreground blocks and weed out the unqualified blocks. Finally, different compensation strategies for the camera and object motions are applied to interpolate the new frames. Illustrative examples are provided to demonstrate the efficacy of the proposed approach. Experimental results are compared with the popular block and non-block based frame interpolation approaches.

Confidence based active learning for whole object image segmentation

In selective object segmentation, the goal is to extract the entire object of interest without regards to homogeneous regions or object shape. In this paper we present the selective image segmentation problem as a classification problem, and use active learning to train an image feature classifier to identify the object of interest. Since our formulation of this segmentation problem uses human interaction, active learning is used for training to minimize the training effort needed to segment the object. Results using several images with known ground truth are presented to show the efficacy of our approach for segmenting the object of interest in still images. The approach has potential applications in medical image segmentation and content-based image retrieval among others.

Dynamically Emerging Semantics in an MPEG-7 Image Database

We examine the efficacy of using MPEG-7 in a native XML database environment for managing aerial image data. This application domain exhibits unique challenges in the management of image data which is continually changing, requiring continuous update of the schemas and methods for mining various geographic region properties. We show how MPEG-7 descriptors can be associated with aerial images and their derived features at each stage of the image insertion process, producing a multifaceted integrated representation, which can be used to give the database administrator an indication as to when this representation should be rebuilt, so as to result in improved retrieval behavior. In the absence of specialized indexes for inexact descriptor matching, we also show how MPEG-7 descriptors can be used to speed up this process.

Moving camera moving object segmentation in an MPEG-2 compressed video sequence

In the paper, we addresses the problem of camera and object motion detection in compressed domain. The estimation of camera motion and the moving object segmentation have been widely stated in a variety of context for video analysis, because they are capable of providing essential clues for interpreting high-level semantic meanings of video sequences. A novel compressed domain motion estimation and segmentation scheme is presented and applied in this paper. The proposed algorithm uses MPEG-2 compressed motion vectors to undergo a spatial and temporal interpolation over several adjacent frames. An iterative rejection scheme based upon the affine model is exploited to effect global camera motion detection. The foreground spatiotemporal objects are separated from the background using the temporal consistency check to the output of the iterative segmentation. This consistency check process can help conglomerate the resulting foreground blocks and weed out unqualified blocks. Illustrative examples are provided to demonstrate the efficacy of the proposed approach.

Video data management using a seeded region growing technique

Video shot partition is the first step towards content-based video analysis. The objective of this paper is to present a novel solution to this problem in detail. This solution is provided in the form of a combination of both spatial and temporal information that are interrelated to each other in video sequences. We model the required measure functions using a robust metric for visual and temporal content similarity and employ a seeded region growing based method to the visual and temporal features of video sequences. The experimental results are demonstrated regarding various transitions and the improvements are shown.

MOVING CAMERA MOVING OBJECT SEGMENTATION IN COMPRESSED VIDEO SEQUENCES

In this paper, we address the problem of camera and object motion detection in the compressed domain. The estimation of camera motion and the moving object segmentation have been widely stated in a variety of context for video analysis, due to their capabilities of providing essential clues for interpreting the high-level semantics of video sequences. A novel compressed domain motion estimation and segmentation scheme is presented and applied in this paper. MPEG-2 compressed domain information, namely Motion Vectors (MV) and Discrete Cosine Transform (DCT) coefficients, is filtered and manipulated to obtain a dense and reliable Motion Vector Field (MVF) over consecutive frames. An iterative segmentation scheme based upon the generalized affine transformation model is exploited to effect the global camera motion detection. The foreground spatiotemporal objects are separated from the background using the temporal consistency check to the output of the iterative segmentation. This consistency check process can coalesce the resulting foreground blocks and weed out unqualified blocks. Illustrative examples are provided to demonstrate the efficacy of the proposed approach.

Video frame rate up conversion using region based motion compensation

Video frame rate conversion is often recognized as video temporal interpolation. In this paper, a region-based approach to automated video frame rate conversion is presented. The region extraction is performed by spatially dividing the reference video frames. Since this step involves image segmentation, the resulting regions are under or over segmented. Consistent frame to frame region extraction is key to region based motion compensation. We propose a novel inter- and intra-frame region merging technique to compensate for inconsistent frame segmentation problem. The video frame rate is then up converted using the motion compensation and region interpolation between every pair of matching regions. The deformation parameters of the match region pair are estimated based on a non-linear estimation model. Heuristic techniques are used to eliminate overlapped and uncovered areas in the generated frames. Experimental results are presented to evaluate its strength over the popular block matching algorithm (BMA).

Work in progress - learning via gaming: An immersive environment for teaching kids handwriting

Immersive learning using computer animation and simulation is an attractive concept. Use of immersive technology to deliver flying lessons to NASA astronauts is well known. While the efficacy of immersive environments for education and training is well established, their accessibility to elementary and middle schools students is negligible. Until recently, hardware costs and long development-times were two major factors impeding creation of such environments for younger students. Computing technology, specifically human machine interface development, has come a long way in the last decade. The authors are making use of this emerging technology to develop an immersive gaming environment to teach handwriting to elementary school children using a tablet PC delivery system.

Video frame rate up conversion under inconsistent camera

In this paper, we address the problem of video frame rate up conversion (FRC) in compressed domain. FRC is often recognized as video temporal interpolation. The problem is very challenging when targeted for a video sequence with an inconsistent camera and object motion. A novel compressed domain motion compensation scheme is presented and applied in this paper. The proposed algorithm uses MPEG-2 compressed motion vectors to undergo a cumulative spatiotemporal interpolation over a temporal sliding window of frames. An iterative rejection scheme based on the affine motion model is exploited to detect the global camera motion. Subsequently, the foreground object separation is performed by examining the temporal consistency of the output of iterative rejections. This consistency check process helps to conglomerate the resulting foreground macroblocks and weeds out the unqualified blocks, thus further refines the crude segmentation results. Finally, different strategies for compensating the camera motion and the object motion are applied to interpolate the new frames. Illustrative examples are provided to demonstrate the efficacy of the proposed approach. Experimental results are compared with the popular block based frame interpolation approach.