O.A. Niamut

MPEG DASH SRD: spatial relationship description

This paper presents the Spatial Representation Description (SRD) feature of the second amendment of MPEG DASH standard part 1, 23009-1:2014 [1]. SRD is an approach for streaming only spatial sub-parts of a video to display devices, in combination with the form of adaptive multi-rate streaming that is intrinsically supported by MPEG DASH. The SRD feature extends the Media Presentation Description (MPD) of MPEG DASH by describing spatial relationships between associated pieces of video content. This enables the DASH client to select and retrieve only those video streams at those resolutions that are relevant to the user experience. The paper describes the design principles behind SRD, the different possibilities it enables and examples of how SRD was used in different experiments on interactive streaming of ultra-high resolution video.

Spatial segmentation for immersive media delivery

Within the EU FP7 project FascinatE, a capture, production and delivery system capable of allowing end-users to interactively view and navigate around an ultra-high resolution video panorama showing a live event is being developed. Network-based processing is used to repurpose the content to suit delivery towards different device types and user selection of regions of interest. In this paper we report on the ongoing developments of the FascinatE delivery network functionality. The associated prototype demonstrates the key concepts of this functionality, showing how spatial segmentation and tiled streaming can be harnessed to allow for efficient content delivery while maintaining high interactivity.

IPTV inter-destination synchronization: A network-based approach

This paper introduces a novel network-based approach to inter-destination media synchronization. The approach meets the need for synchronization in advanced TV concepts like social TV and offers high scalability, unlike conventional end-point based approaches. The solution for inter-destination media synchronization has been standardized in upcoming ETSI protocol standards for IPTV. This paper explains the inner workings of this standardized solution.

Using MPEG DASH SRD for zoomable and navigable video

This paper presents a video streaming client implementation that makes use of the Spatial Relationship Description (SRD) feature of the MPEG-DASH standard, to provide a zoomable and navigable video to an end user. SRD allows a video streaming client to request spatial subparts of a particular video stream, which might be available in multiple resolutions. The paper outlines the design choices enabling the player to render DASH content supporting the SRD feature, such as (i) identifying the total amount of resolution layers and selecting the most appropriate one for the users current selection (pan or zoom), and (ii) enabling a seamless switch between spatial subparts. Doing so, we provide practical implementation guidelines for applications and services that may want to use the SRD feature of MPEG DASH to provide zoomable and navigable video to end users. The video streaming client is implemented in JavaScript and extends dash.js, an MPEG DASH reference client implementation.

Subband merging in cosine-modulated filter banks

Recently, a new method for constructing nonuniform modulated lapped transforms (MLTs) was introduced, by combining subband filters of a uniform MLT. The design, however, was restricted to combining two or four subband filters only, and no systematic design procedure was given. In this letter, we propose an extension to the above-mentioned method that allows arbitrary numbers of subbands to be combined in a systematic way. We investigate the general case of combining filters in arbitrary cosine-modulated filter banks, and give conditions on how to combine the constituent filters such that the resulting nonuniform filter banks have suitable frequency responses.

Advanced interactive television services require content synchronization

Advanced interactive television services, e.g. using IMS-based IPTV technology, enable users to interact with other users within the context of simultaneously consumed content, like broadcast television channels. Differences of content arrival time of 100 ms may already have a perceivable effect on such interaction. Measurements presented in this paper show that these differences can be orders of magnitudes larger in practical systems. The main sources of delay are coding and decoding delays, but also multicasting can introduce significant delay differences. The paper concludes with architectural approaches to inter-destination content synchronization, which can be network- and terminal based. Standardization will be needed in order to make content synchronization solution interoperable between different vendors and service providers.

Towards a format-agnostic approach for production, delivery and rendering of immersive media

The media industry is currently being pulled in the often-opposing directions of increased realism (high resolution, stereoscopic, large screen) and personalization (selection and control of content, availability on many devices). We investigate the feasibility of an end-to-end format-agnostic approach to support both these trends. In this paper, different aspects of a format-agnostic capture, production, delivery and rendering system are discussed. At the capture stage, the concept of layered scene representation is introduced, including panoramic video and 3D audio capture. At the analysis stage, a virtual director component is discussed that allows for automatic execution of cinematographic principles, using feature tracking and saliency detection. At the delivery stage, resolution-independent audiovisual transport mechanisms for both managed and unmanaged networks are treated. In the rendering stage, a rendering process that includes the manipulation of audiovisual content to match the connected display and loudspeaker properties is introduced. Different parts of the complete system are revisited demonstrating the requirements and the potential of this advanced concept.

Optimal time segmentation for overlap-add systems with variable amount of window overlap

In this letter, we propose a new best basis search algorithm for computing the optimal time segmentation of a signal, given a predefined cost measure. The new algorithm solves a problem that arises when the individual signal segments are windowed and overlap-add is applied between adjacent signal segments. When windows having a variable tail shape are employed, the minimization of a cost measure is faced with dependencies between segmental costs due to varying window overlap. A dynamic programming-based algorithm is presented that takes into account these dependencies. It computes both the optimal split positions and the optimal amount of window overlap at these split positions in polynomial time. The proposed algorithm gives an upper bound to the achievable performance of existing algorithms. Experimental results for a modified discrete cosine transform-based processing system are presented, both for entropy and rate-distortion cost measures. These results show a performance gain over existing schemes at the cost of an increased computational complexity.

Flexible frequency decompositions for cosine-modulated filter banks

We investigate the use of nonuniform cosine-modulated filter banks for audio coding. A rate-distortion framework is employed, similar to the work in Herley et al. (1994), to select the filter bank structure from a large library of possible frequency decompositions. A new flexible frequency decomposition algorithm is proposed that jointly optimizes the filter bank structure and the bit allocation over the subband channels. Experimental results for both synthetic and real audio signals are provided. The new algorithm shows significant improvements in comparison with fixed uniform frequency decompositions, but special care has to be taken to reduce the size of the decomposition overhead.

Media Production, Delivery and Interaction for Platform Independent Systems: Format-Agnostic Media

The underlying audio and video processing technology that is discussed in the book relates to areas such as 3D object extraction, audio event detection; 3D sound rendering and face detection, gesture analysis and tracking using video and depth information. The book will give an insight into current trends and developments of future media production, delivery and reproduction. Consideration of the complete production, processing and distribution chain will allow for a full picture to be presented to the reader. Production developments covered will include integrated workflows developed by researchers and industry practitioners as well as capture of ultra-high resolution panoramic video and 3D object based audio across a range of programme genres. Distribution developments will include script based format agnostic network delivery to a full range of devices from large scale public panoramic displays with wavefield synthesis and ambisonic audio reproduction to ’small screen’ mobile devices. Key developments at the consumer end of the chain apply to both passive and interactive viewing modes and will incorporate user interfaces such as gesture recognition and ‘second screen’ devices to allow manipulation of the audio visual content.