P. van Beek

Hierarchical 2-D mesh representation, tracking, and compression for object-based video

This paper proposes methods for designing, tracking and coding hierarchical two-dimensional (2-D) content-based mesh representations. The design procedure consists of constructing a fine-to-coarse hierarchy of Delaunay meshes, using image- and shape-based criteria for mesh geometry simplification. Hierarchical tracking employs a coarse-to-fine strategy with mesh-based motion vector optimization. We introduce new techniques to maintain the initial mesh hierarchy and topology during tracking by imposing certain constraints at each stage of the procedure. The hierarchical compression technique is based on a nearest neighbor ordering of mesh node points. This ordering serves to identify the mesh boundary nodes as well as establish spatial predictors for differential coding of node coordinates and motion vectors. The proposed hierarchical mesh representation, which has applications in object-based video manipulation, investing, and compression, provides improved tracking performance (compared to a nonhierarchical representation) and allows progressive (scalable) transmission of the object geometry (including shape) and motion information, as well as variable level-of-detail rendering. Experimental results are presented to compare the tracking and compression performance of hierarchical versus nonhierarchical mesh representations and to demonstrate the tradeoff between image quality and mesh bit rate for 2-D mesh-based video object rendering.

The MPEG-7 colour structure descriptor: image description using colour and local spatial information

Among the chief applications of the MPEG-7 visual descriptors is image indexing and retrieval. And among the colour image descriptors in the standard having the best retrieval performance is the colour structure (CS) descriptor, a descriptor that uses no more storage than an ordinary colour histogram but that substantially out-performs it. The basis of the CS descriptor is the CS histogram, a generalisation of the colour histogram. The CS histogram encodes information about the spatial structure of the colours in an image as well as their frequency of occurrence. Performance is further enhanced by extracting the CS histogram in the novel nonlinear HMMD colour space and by non-uniformly quantising histogram amplitudes to reflect the bin amplitude statistics derived from consumer oriented image databases. The CS descriptor is scalable in the sense that smaller, more compact, CS descriptors can be easily derived from the largest, highest performance descriptor without re-extraction from the image.

Delay- constrained rate adaptation for robust video transmission over home networks

In this paper, we consider rate adaptation for robust transmission of high-quality video over home networks, in particular IEEE 802.11 wireless LANs. Our approach includes on-line estimation of the time-varying bandwidth available at the application-/transport-layer. We propose a delay-constrained rate adaptation algorithm to select an optimal bit rate, subsequently enforced by a video transcoder. We examine the effectiveness of our novel delay-constrained rate adaptation algorithm, utilizing a simulation environment based on NS-2. Our simulation results show that transmission of MPEG-2 video over an IEEE 802.11b channel results in significantly better quality with rate adaptation compared to transmission without rate adaptation, and that delay-constrained rate adaptation performs significantly better than rate adaptation without a delay constraint.

2-D mesh geometry and motion compression for efficient object-based video representation

Methods for object-based compression and composition of natural and synthetic video content are currently emerging in standards such as MPEG-4 and VRML. This paper describes novel techniques for compression of 2-D triangular mesh geometry and motion, enabling efficient representation and manipulation of video content. Specifically, mesh geometry is compressed by predictive coding of mesh node locations. Mesh node motion vectors are compressed by predictive techniques as well. Preliminary results show that the mesh data can be coded at a fraction of the bits used to code a typical video object.

Optimized transcoding rate selection and packet scheduling for transmitting multiple video streams over a shared channel

We consider the problem of distributed packet selection and scheduling for multiple video streams sharing a communication channel. An optimization framework is proposed, which enables the multiple senders to coordinate their packet transmission schedules, such that the average quality over all video clients is maximized. The framework relies on rate-distortion information that is used to characterize a video packet. This information consists of two quantities: the size of the packet in bits, and its importance for the reconstruction quality of the corresponding stream. A distributed streaming strategy then allows for trading off rate and distortion, not only within a single video stream, but also across different streams. Each of the senders allocates to its own video packets a share of the available bandwidth on the channel in proportion to their importance. We evaluate the performance of the distributed packet scheduling algorithm for two canonical problems in streaming media, namely adaptation to available bandwidth and adaptation to packet loss through prioritized packet retransmissions. Simulation results demonstrate that, for the difficult case of scheduling nonscalably encoded video streams, our framework is very efficient in terms of video quality, both over all streams jointly and also over the individual videos. Compared to a conventional streaming system that does not consider the relative importance of the video packets, the gains in performance range up to 6 dB for the scenario of bandwidth adaptation, and even up to 10 dB for the scenario of random packet loss adaptation.We consider the transmission of multiple video streams from one server to multiple clients over a shared channel. Transcoders are used to dynamically adapt encoded source rates as link capacities vary. We present a closed-form solution for determining optimal values for the transcoded source rates in this case where multiple video streams with independently varying link capacities share time on a channel. We also present a real-time algorithm for optimized packet transmission scheduling and source pruning. Experimental results show gains of 2.5 dB in PSNR when our optimized pruning and scheduling technique is added to a system that uses transcoding, in the case when there is time lag in the control of transcoders. We also show results in which our pruning scheme performs within 1 dB of off-line, R-D optimal pruning.

Spatio-temporal video interpolation and denoising using motion-assisted steering kernel (MASK) regression

In this paper, we extend a (2-D) data-adaptive steering kernel regression framework for image processing to a (3-D) spatio-temporal framework for processing video. In particular, we propose a motion- assisted steering kernel (MASK) suitable for interpolating video data spatially, temporally, or spatio-temporally, and for video noise reduction. We present an algorithm for multi-frame interpolation and reconstruction of video data, and present several simulation results on synthetic and real video data. Comparisons between single-frame and multi-frame kernel regression and with other methods demonstrate the effectiveness of our approach.

Distributed Channel Time Allocation and Rate Adaptation for Multi-User Video Streaming Over Wireless Home Networks

Simultaneous support of multiple video streaming sessions over a shared wireless network requires careful resource allocation to achieve high utilization while dynamically adapting to network and video fluctuations. We propose a distributed algorithm for channel time allocation among multiple video streams, and investigate several heuristic packet pruning schemes for rate adaptation of high-definition (HD) video streams. Simulation results are presented for streaming multiple HD video sequences over an 802.11a network. In comparison with TCP-Friendly Rate Control (TFRC) and a basic scheme without rate adaptation, it is shown that the proposed scheme can sustain higher video quality with lower packet delivery delay.

Multimedia content recommendation engine with automatic inference of user preferences

We propose novel algorithms for automatically determining a users profile from his/her content usage history (profiling agent), and automatically filtering content according to the users profile (filtering agent). A fuzzy inference system is used to construct and periodically update the preferences of a user based on the users interactions with various types of content over an observation period. The proposed algorithms are designed to support an MPEG-7 or TV-anytime-compliant description framework, although they can also be utilized in any non-standard environment that provides structured descriptions of multimedia content.

Optimal hierarchical design of 2D dynamic meshes for videos

This paper proposes methods for designing hierarchical 2D dynamic meshes, for representation of object-based video. This representation consists of a hierarchy of Delaunay meshes, obtained by recursive simplification of the initial fine level-of-detail mesh geometry. Nodes in the initial fine level-of-detail mesh are selected using an edge and corner detector. A dynamic programming-like approach is employed in the mesh simplification procedure to obtain an optimal hierarchical design. Mesh simplification entails removal of mesh nodes to reduce the level of detail. The selection of nodes to be removed is achieved by associating a cost with each mesh node. The hierarchical mesh representation can be applied in object-based video coding, storage and manipulation.

Special issue on analysis and understanding for video adaptation

Presents an introduction to a special issue on the analysis and understanding of video adaptation. The papers in the special issue are very briefly described.