Davy De Schrijver

Using bitstream structure descriptions for the exploitation of multi-layered temporal scalability in H.264/AVC’s base specification

In this paper, attention is paid to the automatic generation of XML-based descriptions containing information about the high-level structure of binary multimedia resources. These structural metadata can then be transformed in order to reflect a desired adaptation of a multimedia resource, and can subsequently be used to create a tailored version of the resource in question. Based on this concept, two technologies are presented: MPEG-21 BSDL and a modified version of XFlavor being able to create BSDL compatible output. Their usage is elaborated in more detail with respect to the valid exploitation of multi-layered temporal scalability in H.264/MPEG-4 AVC’s base specification, and in particular with a focus on a combined usage of the sub-sequence coding technique and Supplemental Enhancement Information (SEI) messages. Some performance measurements in terms of file sizes and computational times are presented as well.

An optimized MPEG-21 BSDL framework for the adaptation of scalable bitstreams

A format-agnostic framework for content adaptation allows reaching a maximum number of users in heterogeneous multimedia environments. Such a framework typically relies on the use of scalable bitstreams. In this paper, we investigate the use of bitstreams compliant with the scalable extension of the H.264/MPEG-4 AVC standard in a format-independent framework for content adaptation. These bitstreams are scalable along the temporal, spatial, and SNR axis. To adapt these bitstreams, a format-independent adaptation engine is employed, driven by the MPEG-21 Bitstream Syntax Description Language (BSDL). MPEG-21 BSDL is a specification that allows generating high-level XML descriptions of the structure of a scalable bitstream. As such, the complexity of the adaptation of scalable bitstreams can be moved to the XML domain. Unfortunately, the current version of MPEG-21 BSDL cannot be used to describe the structure of large video bitstreams because the bitstream parsing process is characterized by an increasing memory consumption and a decreasing description generation speed. Therefore, in this paper, we describe a number of extensions to the MPEG-21 BSDL specification that make it possible to optimize the processing of bitstreams. Moreover, we also introduce a number of additional extensions necessary to describe the structure of scalable H.264/AVC bitstreams. Our performance analysis demonstrates that our extensions enable the bitstream parsing process to translate the structure of the scalable bitstreams into an XML document multiple times faster. Further, a constant and low memory consumption is obtained during the bitstream parsing process.

A real-time content adaptation framework for exploiting ROI scalability in H.264/AVC

It is well known that motion detection using single frame differencing, while computationally much simpler than other techniques, is more liable to generate large areas of false foregrounds known as ghosts. In order to overcome this problem the authors propose a method based on signed differencing and connectivity analysis. The proposal is suitable to applications which cannot afford the un-avoidable errors of background modeling or the limitations of 3-frames preprocessing.In many application scenarios, the use of Regions of Interest (ROIs) within video sequences is a useful concept. It is shown in this paper how Flexible Macroblock Ordering (FMO), defined in H.264/AVC as an error resilience tool, can be used for the coding arbitrary-shaped ROIs. In order to exploit the coding of ROIs in an H.264/AVC bitstream, a description-driven content adaptation framework is introduced that is able to extract the ROIs of a given bitstream. The results of a series of tests indicate that the ROI extraction process significantly reduces the bit rate of the bitstreams and increases the decoding speed. In case of a fixed camera and a static background, the impact of this reduction on the visual quality of the video sequence is negligible. Regarding the adaptation framework itself, it is shown that in all cases, the framework operates in real time and that it is suited for streaming scenarios by design.

BFlavor: A harmonized approach to media resource adaptation, inspired by MPEG-21 BSDL and XFlavor

During recent years, several tools have been developed that allow the automatic generation of XML descriptions containing information about the syntax of binary media resources. Such a bitstream syntax description (BSD) can then be transformed to reflect a desired adaptation of a media resource, and can subsequently be used to create a tailored version of this resource. The main contribution of this paper is the introduction of BFlavor, a new tool for exposing the syntax of binary media resources as an XML description. Its development was inspired by two other technologies, i.e. MPEG-21 BSDL and XFlavor. Although created from a different point of view, both languages offer solutions for translating the syntax of a media resource into an XML representation for further processing. BFlavor (BSDL+XFlavor) harmonizes the two technologies by combining their strengths and eliminating their weaknesses. More precisely, the processing efficiency and expressive power of XFlavor on the one hand, and the ability to create high-level BSDs using MPEG-21 BSDL on the other hand, were our key motives for its development. To assess the expressive power and performance of a BFlavor-driven content adaptation chain, several experiments were conducted. These experiments test the automatic generation of BSDs for MPEG-1 Video and H.264/AVC, as well as the exploitation of multi-layered temporal scalability in H.264/AVC. Our results show that BFlavor is an efficient and harmonized description tool for enabling XML-driven adaptation of media resources in a format-agnostic way. BSDL and XFlavor are outperformed by BFlavor in terms of execution times, memory consumption, and file sizes.

MPEG-21 bitstream syntax descriptions for scalable video codecs

In order to obtain a useful multichannel publication environment, a content producer has to respect the different terminal and network characteristics of the multimedia devices of its target audience. Embedded scalable video bitstreams, together with a complementary content adaptation framework, give the possibility to respond to heterogeneous usage environments. In this paper, temporally scalable H.264/MPEG-4 AVC encoded bitstreams and bitstreams encoded by relying on the fully-embedded MC-EZBC wavelet-based codec are used. The MPEG-21 Bitstream Syntax Description Language (BSDL) specification is used to generate high-level XML descriptions of the structure of a bitstream. As such, the adaptation of a scalable video stream can be realized in the XML domain, rather than on the bitstream itself. Different transformation technologies are compared to each other as well. Finally, a practical setup of a video streaming use case is discussed by relying on the MPEG-21 BSDL framework.

Scalable, Wavelet-Based Video: From Server to Hardware-Accelerated Client

Video source, carrier and client diversification have led the video coding community to develop scalable video codecs supporting efficient decoding at varying resolution, frame rate and quality. Scalable video has several advantages over a nonscalable approach, but a large scale deployment is far from trivial and a lot of open questions remain. To resolve these, we developed a complete video delivery chain for scalable wavelet-based video. This includes a video server, a negotiation framework, a video scaling infrastructure and two scalable video clients, one pure software client and one real-time, hardware accelerated client. This paper describes the complete chain and identifies and quantifies the impact of using scalable video in every link of this chain.

Generating MPEG-21 BSDL descriptions using context-related attributes

In order to efficiently deal with the heterogeneity in the current and future multimedia ecosystem, it is necessary that content can be adapted in a format-agnostic manner. A first step toward a solution, able to fulfill the just mentioned requirement, is to rely on a scalable video codec and to describe the high-level structure of the resulting bitstreams in such a way that every terminal can understand it, in particular by using XML. This paper describes how such descriptions can be generated by making use of the media format independent BintoBSD tool of the MPEG-21 BSDL standard. However, regarding the current status of BSDL, it is impossible to create a description in real time and to keep the generation speed constant over the complete sequence. In this paper, we describe a number of extensions and algorithmic modifications that make it possible to generate a description of a bitstream in real time and at a constant speed. Our approach results in a significant reduction of the original execution times (up to 99% for the H.264/AVC coding format) and in a constant memory usage.

Is That a Fish in Your Ear? A Universal Metalanguage for Multimedia

Developing the code to parse and generate multimedia bitstreams has traditionally been a repetitive and error-prone task. It has also been an area of application development that defied the goal of software reuse. In contrast, BSDL abstracts the minutiae of bitstream parsing out of software code, into an interoperable data file (the BSDL schema), allowing developers to concentrate on the functionality of their particular application. BSDLs approach has demonstrated applications at numerous points in the multimedia delivery chain. In the future, this approach may be extended to still other processing tasks, such as transcoding and transmoding, or to types of binary data other than multimedia

Enhanced Shot-Based Video Adaptation using MPEG-21 generic Bitstream Syntax Schema

Semantic video adaptation takes into account the relevance of the different fragments of the video content in order to create a tailored video stream based on the users preferences. As a shot can be considered as the smallest semantic unit in a video sequence, metadata can be added to each shot using MPEG-7 descriptions. Based on these metadata and the users preferences, the original bitstream can be adapted in order to obtain the desired fragments. MPEG-21 DIA offers a tool, gBS schema, for exposing the high-level structure of a binary resource as an XML description. In this paper, shot information is inserted in these descriptions to create a link between metadata and semantic video adaptation. Furthermore, this paper proposes to keep the structure of these descriptions format-agnostic. As a result, only one generic transformation style sheet has to be implemented to support shot-based video adaptation of sequences compliant with different video specifications. Special attention is paid to sequences coded with the H.264/AVC standard as this specification contains several interesting features important for shot-based video adaptation

A performance evaluation of the data partitioning tool in H.264/AVC

In order to be able to better cope with packet loss, H.264/AVC, besides offering superior coding efficiency, also comes with a number of error resilience tools. The goal of these tools is to enable the decoding of a bitstream containing encoded video, even when parts of it are missing. On top of that, the visual quality of the decoded video should remain as high as possible. In this paper, we will discuss and evaluate one of these tools, in particular the data partitioning tool. Experimental results will show that using data partitioning can significantly improve the quality of a video sequence when packet loss occurs. However, this is only possible if the channel used for transmitting the video allows selective protection of the different data partitions. In the most extreme case, an increase in PSNR of up to 9.77 dB can be achieved. This paper will also show that the overhead caused by using data partitioning is acceptable. In terms of bit rate, the overhead amounts to approximately 13 bytes per slice. In general, this is less than 1% of the total bit rate. On top of that, using constrained intra prediction, which is required to fully exploit data partitioning, causes a decrease in quality of about 0.5 dB for high quality video and between 1 and 2 dB for low quality video.