Koen De Wolf

Using bitstream structure descriptions for the exploitation of multi-layered temporal scalability in H.264/AVC’s base specification

In this paper, attention is paid to the automatic generation of XML-based descriptions containing information about the high-level structure of binary multimedia resources. These structural metadata can then be transformed in order to reflect a desired adaptation of a multimedia resource, and can subsequently be used to create a tailored version of the resource in question. Based on this concept, two technologies are presented: MPEG-21 BSDL and a modified version of XFlavor being able to create BSDL compatible output. Their usage is elaborated in more detail with respect to the valid exploitation of multi-layered temporal scalability in H.264/MPEG-4 AVC’s base specification, and in particular with a focus on a combined usage of the sub-sequence coding technique and Supplemental Enhancement Information (SEI) messages. Some performance measurements in terms of file sizes and computational times are presented as well.

Requantization transcoding of H.264/AVC bitstreams for intra 4 x 4 prediction modes

Efficient bitrate reduction of video content is necessary in order to satisfy the different constraints imposed by decoding devices and transmission networks. Requantization is a fast technique for bitrate reduction, and has been successfully applied for MPEG-2 bitstreams. Because of the newly introduced intra prediction in H.264/AVC, the existing techniques are rendered useless. In this paper we examine requantization transcoding of H.264/AVC bitstreams, focusing on the intra 4x4 prediction modes. Two architectures are proposed, one in the pixel domain and the other in the frequency domain, that compensate tlie drift introduced by the requantization of intra 4x4 predicted blocks. Experimental results show that these architectures perform approximately equally well as the full decode and recode architecture for low to medium bitrates. Because of the reduced computational complexity of these architectures, in particular the frequency-domain compensation architecture, they are highly suitable for real-time adaptation of video content.Although no scalability is explicitly defined in the H.264/ AVC specification, some forms of scalability can be achieved by using the available coding tools in a creative way. In this paper we will explain how to use the data partitioning tool to perform a coarse form of SNR scalability. The impact of various parameters, including the presence of IDR frames and the number of intra-coded macroblocks per frame, on bit rate and bit rate savings and on quality and quality loss will be discussed. Furthermore we will introduce and elaborate a possible use case for the technique proposed in this paper.MPEG-21 BSDL offers a solution for exposing the structure of a binary media resource as an XML description, and for the generation of a tailored media resource using a transformed XML description. The main contribution of this paper is the introduction of a real-time work flow for the XML-driven adaptation of H.264/AVC bitstreams in the temporal domain. This real-time approach, which is in line with the vision of MPEG-21 BSDL, is made possible by two key technologies: BFlavor (BSDL + XFlavor) for the efficient generation of XML descriptions and Streaming Transformations for XML (STX) for the efficient transformation of these descriptions. Our work flow is validated in several applications, all using H.264/AVC bitstreams: the exploitation and emulation of temporal scalability, as well as the creation of video skims using key frame selection. Special attention is paid to the deployment of hierarchical B pictures and to the use of placeholder slices for synchronization purposes. Extensive performance data arc also provided.Expressive Speech Recognition and Synthesis as Enabling Technologies for Affective Robot-Child Communication.- Embodied Conversational Agents: Computing and Rendering Realistic Gaze Patterns.- DBN Based Models for Audio-Visual Speech Analysis and Recognition.- An Extensive Method to Detect the Image Digital Watermarking Based on the Known Template.- Fast Mode Decision Algorithm in H.263+/H.264 Intra Transcoder.- Binary Erasure Codes for Packet Transmission Subject to Correlated Erasures.- Image Desynchronization for Secure Collusion-Resilient Fingerprint in Compression Domain.- A Format-Compliant Encryption Framework for JPEG2000 Image Code-Streams in Broadcasting Applications.- Euclidean Distance Transform of Digital Images in Arbitrary Dimensions.- JPEG2000 Steganography Possibly Secure Against Histogram-Based Attack.- Perceptual Depth Estimation from a Single 2D Image Based on Visual Perception Theory.- A System for Generating Personalized Virtual News.- Image Fingerprinting Scheme for Print-and-Capture Model.- 16x16 Integer Cosine Transform for HD Video Coding.- Heegard-Berger Video Coding Using LMMSE Estimator.- Real-Time BSD-Driven Adaptation Along the Temporal Axis of H.264/AVC Bitstreams.- Optimal Image Watermark Decoding.- Diagonal Discrete Cosine Transforms for Image Coding.- Synthesizing Variational Direction and Scale Texture on Planar Region.- Fast Content-Based Image Retrieval Based on Equal-Average K-Nearest-Neighbor Search Schemes.- Characterizing User Behavior to Improve Quality of Streaming Service over P2P Networks.- Interacting Activity Recognition Using Hierarchical Durational-State Dynamic Bayesian Network.- Improving the Image Retrieval Results Via Topic Coverage Graph.- Relevance Feedback for Sketch Retrieval Based on Linear Programming Classification.- Hierarchical Motion-Compensated Frame Interpolation Based on the Pyramid Structure.- Varying Microphone Patterns for Meeting Speech Segmentation Using Spatial Audio Cues.- Region-Based Sub-pixel Motion Estimation from Noisy, Blurred, and Down-Sampled Sequences.- Differential Operation Based Palmprint Authentication for Multimedia Security.- A Broadcast Model for Web Image Annotation.- An Approach to the Compression of Residual Data with GPCA in Video Coding.- A Robust Approach for Object Recognition.- A Novel Method for Spoken Text Feature Extraction in Semantic Video Retrieval.- A Semantic Image Category for Structuring TV Broadcast Video Streams.- Markov Chain Monte Carlo Super-Resolution Image Reconstruction with Simultaneous Adaptation of the Prior Image Model.- Text Detection in Images Using Texture Feature from Strokes.- Robust Mandarin Speech Recognition for Car Navigation Interface.- GKDA: A Group-Based Key Distribution Algorithm for WiMAX MBS Security.- A Watermarking Algorithm for JPEG File.- SNR Scalability in H.264/AVC Using Data Partitioning.- A Real-Time XML-Based Adaptation System for Scalable Video Formats.- Generic, Scalable Multimedia Streaming and Delivery with Example Application for H.264/AVC.- Shape-Based Image Retrieval in Botanical Collections.- Macroblock Mode Decision Scheme for Fast Encoding in H.264/AVC.- A Mathematical Model for Interaction Analysis Between Multiview Video System and User.- Motion Composition of 3D Video.- EKM: An Efficient Key Management Scheme for Large-Scale Peer-to-Peer Media Streaming.- Using Earth Movers Distance for Audio Clip Retrieval.- Streaming-Mode MB-Based Integral Image Techniques for Fast Multi-view Video Illumination Compensation.- A Motion Vector Predictor Architecture for AVS and MPEG-2 HDTV Decoder.- Inter-camera Coding of Multi-view Video Using Layered Depth Image Representation.- Optimal Priority Packetization with Multi-layer UEP for Video Streaming over Wireless Network.- A Multi-channel MAC Protocol with Dynamic Channel Allocation in CDMA Ad Hoc Networks.- Fuzzy Particle Swarm Optimization Clustering and Its Application to Image Clustering.- A New Fast Motion Estimation for H.264 Based on Motion Continuity Hypothesis.- Statistical Robustness in Multiplicative Watermark Detection.- Adaptive Visual Regions Categorization with Sets of Points of Interest.- A Publishing Framework for Digitally Augmented Paper Documents: Towards Cross-Media Information Integration.- Web-Based Semantic Analysis of Chinese News Video.- A Quality-Controllable Encryption for H.264/AVC Video Coding.- Texture Synthesis Based on Minimum Energy Cut and Its Applications.- Unifying Keywords and Visual Features Within One-Step Search for Web Image Retrieval.- Dynamic Bandwidth Allocation for Stored Video Under Renegotiation Frequency Constraint.- Online Selection of Discriminative Features Using Bayes Error Rate for Visual Tracking.- Interactive Knowledge Integration in 3D Cloth Animation with Intelligent Learning System.- Multi-view Video Coding with Flexible View-Temporal Prediction Structure for Fast Random Access.- Squeezing the Auditory Space: A New Approach to Multi-channel Audio Coding.- Video Coding by Texture Analysis and Synthesis Using Graph Cut.- Multiple Description Coding Using Adaptive Error Recovery for Real-Time Video Transmission.- An Improved Motion Vector Prediction Scheme for Video Coding.- Classifying Motion Time Series Using Neural Networks.- Estimating Intervals of Interest During TV Viewing for Automatic Personal Preference Acquisition.- Image Annotations Based on Semi-supervised Clustering with Semantic Soft Constraints.- Photo Retrieval from Personal Memories Using Generic Concepts.- PanoWalk: A Remote Image-Based Rendering System for Mobile Devices.- A High Quality Robust Watermarking Scheme.- An Association Rule Mining Approach for Satellite Cloud Images and Rainfall.- AVAS: An Audio-Visual Attendance System.- Improved POCS-Based Deblocking Technique Using Wavelet Transform in Block Coded Image.- Sketch Case Based Spatial Topological Data Retrieval.- Providing Consistent Service for Structured P2P Streaming System.- Adaptive Search Range Scaling for B Pictures Coding.- Video QoS Monitoring and Control Framework over Mobile and IP Networks.- Extracting Moving / Static Objects of Interest in Video.- Building a Personalized Music Emotion Prediction System.- Video Segmentation Using Joint Space-Time-Range Adaptive Mean Shift.- EagleRank: A Novel Ranking Model for Web Image Search Engine.- Color Image Enhancement Using the Laplacian Pyramid.- 3D Mesh Construction from Depth Images with Occlusion.- An Eigenbackground Subtraction Method Using Recursive Error Compensation.- Attention Information Based Spatial Adaptation Framework for Browsing Videos Via Mobile Devices.- Style Strokes Extraction Based on Color and Shape Information.- Requantization Transcoding of H.264/AVC Bitstreams for Intra 4x4 Prediction Modes.- Prediction Algorithms in Large Scale VOD Services on Grid Infrastructure.- A Hierarchical Framework for Fast Macroblock Prediction Mode Decision in H.264.- Compact Representation for Large-Scale Clustering and Similarity Search.- Robust Recognition of Noisy and Partially Occluded Faces Using Iteratively Reweighted Fitting of Eigenfaces.- Pitching Shot Detection Based on Multiple Feature Analysis and Fuzzy Classification.- Color Changing and Fading Simulation for Frescoes Based on Empirical Knowledge from Artists.- A Novel Spatial-Temporal Position Prediction Motion-Compensated Interpolation for Frame Rate Up-Conversion.- Web Image Clustering with Reduced Keywords and Weighted Bipartite Spectral Graph Partitioning.- An Architecture to Connect Disjoint Multimedia Networks Based on Nodes Capacity.- Quantitative Measure of Inlier Distributions and Contour Matching for Omnidirectional Camera Calibration.- High-Speed All-in-Focus Image Reconstruction by Merging Multiple Differently Focused Images.- A Real-Time Video Deinterlacing Scheme for MPEG-2 to AVS Transcoding.- Persian Text Watermarking.- Three Dimensional Reconstruction of Structured Scenes Based on Vanishing Points.- Parallel Processing for Reducing the Bottleneck in Realtime Graphics Rendering.- Distributed Data Visualization Tools for Multidisciplinary Design Optimization of Aero-crafts.- An Efficient Clustering and Indexing Approach over Large Video Sequences.- An Initial Study on Progressive Filtering Based on Dynamic Programming for Query-by-Singing/Humming.- Measuring Multi-modality Similarities Via Subspace Learning for Cross-Media Retrieval.- SNR-Based Bit Allocation in Video Quality Smoothing.- Shadow Removal in Sole Outdoor Image.- 3D Head Model Classification Using KCDA.- Framework for Pervasive Web Content Delivery.- Region-Based Semantic Similarity Propagation for Image Retrieval.Scalable bitstreams are used today to contribute to the Universal Multimedia Access (UMA) philosophy, i.e., accessing multimedia anywhere, at anytime, and on any device. Bitstream structure description languages provide means to adapt scalable bitstreams in order to extract a lower quality version. This paper introduces a real-time XML-based framework for content adaptation by relying on BFlavor, a combination of two existing bitstream structure description languages (i.e., the MPEG-21 Bitstream Syntax Description Language (BSDL) and the Formal Language for Audio-Visual Representation extended with XML features (XFlavor)). In order to use BFlavor with state-of-the-art media formats, we have added support for transparent retrieval of context information and support for emulation prevention bytes. These extensions are validated by building a BFlavor code for bitstreams compliant with the scalable extension of the H.264/AVC specification. Performance measurements show that such a bitstream (containing a bitrate of 17 MBit/s) can be adapted in real-time by a BFlavor-based adaptation framework (with a speed of 27 MBit/s).

An optimized MPEG-21 BSDL framework for the adaptation of scalable bitstreams

A format-agnostic framework for content adaptation allows reaching a maximum number of users in heterogeneous multimedia environments. Such a framework typically relies on the use of scalable bitstreams. In this paper, we investigate the use of bitstreams compliant with the scalable extension of the H.264/MPEG-4 AVC standard in a format-independent framework for content adaptation. These bitstreams are scalable along the temporal, spatial, and SNR axis. To adapt these bitstreams, a format-independent adaptation engine is employed, driven by the MPEG-21 Bitstream Syntax Description Language (BSDL). MPEG-21 BSDL is a specification that allows generating high-level XML descriptions of the structure of a scalable bitstream. As such, the complexity of the adaptation of scalable bitstreams can be moved to the XML domain. Unfortunately, the current version of MPEG-21 BSDL cannot be used to describe the structure of large video bitstreams because the bitstream parsing process is characterized by an increasing memory consumption and a decreasing description generation speed. Therefore, in this paper, we describe a number of extensions to the MPEG-21 BSDL specification that make it possible to optimize the processing of bitstreams. Moreover, we also introduce a number of additional extensions necessary to describe the structure of scalable H.264/AVC bitstreams. Our performance analysis demonstrates that our extensions enable the bitstream parsing process to translate the structure of the scalable bitstreams into an XML document multiple times faster. Further, a constant and low memory consumption is obtained during the bitstream parsing process.

BFlavor: A harmonized approach to media resource adaptation, inspired by MPEG-21 BSDL and XFlavor

During recent years, several tools have been developed that allow the automatic generation of XML descriptions containing information about the syntax of binary media resources. Such a bitstream syntax description (BSD) can then be transformed to reflect a desired adaptation of a media resource, and can subsequently be used to create a tailored version of this resource. The main contribution of this paper is the introduction of BFlavor, a new tool for exposing the syntax of binary media resources as an XML description. Its development was inspired by two other technologies, i.e. MPEG-21 BSDL and XFlavor. Although created from a different point of view, both languages offer solutions for translating the syntax of a media resource into an XML representation for further processing. BFlavor (BSDL+XFlavor) harmonizes the two technologies by combining their strengths and eliminating their weaknesses. More precisely, the processing efficiency and expressive power of XFlavor on the one hand, and the ability to create high-level BSDs using MPEG-21 BSDL on the other hand, were our key motives for its development. To assess the expressive power and performance of a BFlavor-driven content adaptation chain, several experiments were conducted. These experiments test the automatic generation of BSDs for MPEG-1 Video and H.264/AVC, as well as the exploitation of multi-layered temporal scalability in H.264/AVC. Our results show that BFlavor is an efficient and harmonized description tool for enabling XML-driven adaptation of media resources in a format-agnostic way. BSDL and XFlavor are outperformed by BFlavor in terms of execution times, memory consumption, and file sizes.

Requantization transcoding of H.264/AVC bitstreams for intra 4×4 prediction modes

Efficient bitrate reduction of video content is necessary in order to satisfy the different constraints imposed by decoding devices and transmission networks. Requantization is a fast technique for bitrate reduction, and has been successfully applied for MPEG-2 bitstreams. Because of the newly introduced intra prediction in H.264/AVC, the existing techniques are rendered useless. In this paper we examine requantization transcoding of H.264/AVC bitstreams, focusing on the intra 4×4 prediction modes. Two architectures are proposed, one in the pixel domain and the other in the frequency domain, that compensate the drift introduced by the requantization of intra 4×4 predicted blocks. Experimental results show that these architectures perform approximately equally well as the full decode and recode architecture for low to medium bitrates. Because of the reduced computational complexity of these architectures, in particular the frequency-domain compensation architecture, they are highly suitable for real-time adaptation of video content.

Comparison of prediction schemes with motion information reuse for low complexity spatial scalability

Three low complexity algorithms that allow spatial scalability in the context of video coding are presented in this paper. We discussed the feasibility of reusing motion and residual texture information of the base layer in the enhancement layer. The prediction errors that arise from the discussed filters and schemes are evaluated in terms of the Mean of Absolute Differences. For the interpolation of the decoded pictures from the base layer, the presented 6-tap and bicubic filters perform significantly better than the bilinear and nearest neighbor filters. In contrast, when reusing the motion vector field and the error pictures of the base layer, the bilinear filter performs best for the interpolation of residual texture information. In general, reusing the motion vector field and the error pictures of the base layer gives the lowest prediction errors. However, our tests showed that for some sequences that have regions with complex motion activity, interpolating the decoded picture of the base layer gives best result. This means that an encoder should compare all possible prediction schemes combined with all interpolation filters in order to achieve optimal prediction. Obviously this would not be possible for real-time content creation.Three low complexity algorithms that allow spatial scalability in the context of video coding are presented in this paper. We discussed the feasibility of reusing motion and residual texture information of the base layer in the enhancement layer. The prediction errors that arise from the discussed filters and schemes are evaluated in terms of the Mean of Absolute Differences. For the interpolation of the decoded pictures from the base layer, the presented 6-tap and bicubic filters perform significantly better than the bilinear and nearest neighbor filters. In contrast, when reusing the motion vector field and the error pictures of the base layer, the bilinear filter performs best for the interpolation of residual texture information. In general, reusing the motion vector field and the error pictures of the base layer gives the lowest prediction errors. However, our tests showed that for some sequences that have regions with complex motion activity, interpolating the decoded picture of the base layer gives best result. This means that an encoder should compare all possible prediction schemes combined with all interpolation filters in order to achieve optimal prediction. Obviously this would not be possible for real-time content creation.

Performance evaluation of adaptive residual interpolation, a tool for inter-layer prediction in H.264/AVC scalable video coding

Inter-layer prediction is the most important technique for improving coding performance in spatial enhancement layers in Scalable Video Coding (SVC). In this paper we discuss Adaptive Residual Interpolation (ARI), a new approach to inter-layer prediction of residual data. This prediction method yields a higher coding performance. We integrated the ARI tool in the Joint Scalable Video Model software. Special attention was paid to the CABAC context model initialization. Further, the use, complexity, and coding performance of this technology is discussed. Three filters were tested for the interpolation of lower-layer residuals: a bi-linear filter, the H.264/AVC 6-tap filter, and a median filter. Tests have shown that ARI prediction results in an average bit rate reduction of 0.40% for the tested configurations without a loss in visual quality. In a particular test case, a maximum bit rate reduction of 10.10% was observed for the same objective quality.

Quality analysis of requantization transcoding architectures for H.264/AVC

Reduction of the bitrate of video content is necessary in order to satisfy the different constraints imposed by networks and terminals. A fast and elegant solution for the reduction of the bitrate is requantization, which has been successfully applied on MPEG-2 bitstreams. Because of the improved intra prediction in the H.264/AVC specification, existing transcoding techniques are no longer suitable. In this paper we compare requantization transcoders for H.264/AVC bitstreams. The discussion is restricted to intra 4x4 macroblocks only, but the same techniques are also applicable to intra 16x16 macroblocks. Besides the open-loop transcoder and the transcoder with mode reuse, two architectures with drift compensation are described, one in the pixel domain and the other in the transform domain. Experimental results show that these architectures approach the quality of the full decode and recode architecture for low to medium bitrates. Because of the reduced computational complexity of these architectures, in particular the transform-domain compensation architecture, they are highly suitable for real-time adaptation of video content.