Miska Hannuksela

The emerging MVC standard for 3D video services

Multiview video has gained a wide interest recently. The huge amount of data needed to be processed by multiview applications is a heavy burden for both transmission and decoding. The joint video team has recently devoted part of its effort to extend the widely deployed H.264/AVC standard to handle multiview video coding (MVC). The MVC extension of H.264/AVC includes a number of new techniques for improved coding efficiency, reduced decoding complexity, and new functionalities for multiview operations. MVC takes advantage of some of the interfaces and transport mechanisms introduced for the scalable video coding (SVC) extension of H.264/AVC, but the system level integration of MVC is conceptually more challenging as the decoder output may contain more than one view and can consist of any combination of the views with any temporal level. The generation of all the output views also requires careful consideration and control of the available decoder resources. In this paper, multiview applications and solutions to support generic multiview as well as 3D services are introduced. The proposed solutions, which have been adopted to the draft MVC specification, cover a wide range of requirements for 3D video related to interface, transport of the MVC bitstreams, and MVC decoder resource management. The features that have been introduced in MVC to support these solutions include marking of reference pictures, supporting for efficient view switching, structuring of the bitstream, signalling of view scalability supplemental enhancement information (SEI) and parallel decoding SEI.

H.264/AVC video for wireless transmission

H.264/AVC will be an essential component in emerging wireless video applications thanks to its excellent compression efficiency and network-friendly design. However, a video coding standard itself is only one component within the application and transmission environment. Its effectiveness strongly depends on the selection of appropriate modes and parameters at the encoder, at the decoder, as well as in the network. In this paper we introduce the features of the H.264/AVC coding standard that make it suitable for wireless video applications, including features for error resilience, bit rate adaptation, integration into packet networks, interoperability, and buffering considerations. Modern wireless networks provide many different means to adapt quality of service, such as forward error correction methods on different layers and end-to-end or link layer retransmission protocols. The applicability of all these encoding and network features depends on application constraints, such as the maximum tolerable delay, the possibility of online encoding, and the availability of feedback and cross-layer information. We discuss the use of different coding and transport related features for different applications, namely video telephony, video conferencing, video streaming, download-and-play, and video broadcasting. Guidelines for the selection of appropriate video coding tools, video encoder and decoder settings, as well as transport and network parameters are provided and justified. References to relevant research publications and standardization contributions are given.

Overview of HEVC High-Level Syntax and Reference Picture Management

The increasing proportion of video traffic in telecommunication networks puts an emphasis on efficient video compression technology. High Efficiency Video Coding (HEVC) is the forthcoming video coding standard that provides substantial bit rate reductions compared to its predecessors. In the HEVC standardization process, technologies such as picture partitioning, reference picture management, and parameter sets are categorized as “high-level syntax.” The design of the high-level syntax impacts the interface to systems and error resilience, and provides new functionalities. This paper presents an overview of the HEVC high-level syntax, including network abstraction layer unit headers, parameter sets, picture partitioning schemes, reference picture management, and supplemental enhancement information messages.

System and Transport Interface of SVC

Scalable video coding (SVC) and transmission has been a research topic for many years. Among other objectives, it aims to support different receiving devices, perhaps connected through a heterogeneous network structure, using a single bit stream. Earlier attempts of standardized scalable video coding, for example in MPEG-2, H.263, or MPEG-4 Visual, have not been commercially successful. Nevertheless, the Joint Video Team has recently focused on the development of the scalable video extensions of H.264/AVC, known as SVC. Some of the key problems of older scalable compression techniques have been solved in SVC and, at the same time, new and compelling use cases for SVC have been identified. While it is certainly important to develop coding tools targeted at high coding efficiency, the design of the features of the interface between the core coding technologies and the system and transport are also of vital importance for the success of SVC. Only through this interface, and novel mechanisms defined therein, applications can take advantage of the scalability features of the coded video signal. This paper provides an overview of the system interface features defined in the SVC specification. We discuss, amongst other features, bit stream structure, extended network abstraction layer (NAL) unit header, and supplemental enhancement information (SEI) messages related to scalability information.

Perceptual-based quality assessment for audio-visual services: A survey

Accurate measurement of the perceived quality of audio-visual services at the end-user is becoming a crucial issue in digital applications due to the growing demand for compression and transmission of audio-visual services over communication networks. Content providers strive to offer the best quality of experience for customers linked to their different quality of service (QoS) solutions. Therefore, developing accurate, perceptual-based quality metrics is a key requirement in multimedia services. In this paper, we survey state-of-the-art signal-driven perceptual audio and video quality assessment methods independently, and investigate relevant issues in developing joint audio-visual quality metrics. Experiments with respect to subjective quality results have been conducted for analyzing and comparing the performance of the quality metrics. We consider emerging trends in audio-visual quality assessment, and propose feasible solutions for future work in perceptual-based audio-visual quality metrics.

Does context matter in quality evaluation of mobile television

Subjective quality evaluation is used to optimize the produced audiovisual quality from fundamental signal processing algorithms to consumer services. These studies typically follow the basic principles of controlled psychoperceptual experiments. However, when compromising compression and transmission parameters for consumer services, the ecological validity of conventional quality evaluation methods can be questioned. To tackle this, we firstly present a novel user-oriented quality evaluation method for mobile television in its usage contexts. Secondly, we present the results of an experiment conducted with 30 participants comparing acceptability and satisfaction of quality as well as goals of viewing in three mobile contexts and under four different residual transmission error rates, when the participants also performed simultaneous assessment tasks. Finally, we compare the results with a previous laboratory experiment. The studied error rates impacted negatively on all measured tasks with some contextual differences. Moreover, the evaluations were more favorable and less discriminate in the mobile contexts compared to the laboratory.

System Layer Integration of High Efficiency Video Coding

This paper describes the integration of High Efficiency Video Coding (HEVC) into end-to-end multimedia systems, formats, and protocols such as Real-time transport Protocol, the transport stream of the MPEG-2 standard suite, and dynamic adaptive streaming over the Hypertext Transport Protocol. This paper gives a brief overview of the high-level syntax of HEVC and the relation to the Advanced Video Coding standard (H.264/AVC). A section on HEVC error resilience concludes the HEVC overview. Furthermore, this paper describes applications of video transport and delivery such as broadcast, television over the Internet Protocol, Internet streaming, video conversation, and storage as provided by the different system layers.

Subjective study on compressed asymmetric stereoscopic video

Asymmetric stereoscopic video coding takes advantage of the binocular suppression of the human vision by representing one of the views with a lower quality. This paper describes a subjective quality test with asymmetric stereoscopic video. Different options for achieving compressed mixed-quality and mixed-resolution asymmetric stereo video were studied and compared to symmetric stereo video. The bitstreams for different coding arrangements were simulcast-coded according to the Advanced Video Coding (H.264/AVC) standard. The results showed that in most cases, resolution-asymmetric stereo video with the downsampling ratio of 1/2 along both coordinate axes provided similar quality as symmetric and quality-asymmetric full-resolution stereo video. These results were achieved under same bitrate constrain while the processing complexity decreased considerably. Moreover, in all test cases, the symmetric and mixed-quality full-resolution stereoscopic video bitstreams resulted in a similar quality at the same bitrates.

Perceptual quality assessment based on visual attention analysis

Most existing quality metrics do not take the human attention analysis into account. Attention to particular objects or regions is an important attribute of human vision and perception system in measuring perceived image and video qualities. This paper presents an approach for extracting visual attention regions based on a combination of a bottom-up saliency model and semantic image analysis. The use of PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural SIMilarity) in extracted attention regions is analyzed for image/video quality assessment, and a novel quality metric is proposed which can exploit the attributes of visual attention information adequately. The experimental results with respect to the subjective measurement demonstrate that the proposed metric outperforms the current methods.

Rate adaptation for dynamic adaptive streaming over HTTP in content distribution network

Recently the 3rd Generation Partnership Project (3GPP) and the Moving Picture Experts Group (MPEG) specified Dynamic Adaptive Streaming over HTTP (DASH) to cope with the shortages in progressive HTTP based downloading and Real-time Transport Protocol (RTP) over the User Datagram Protocol (UDP), shortly RTP/UDP, based streaming. This paper investigates rate adaptation for the serial segment fetching method and the parallel segment fetching method in Content Distribution Network (CDN). The serial segment fetching method requests and receives segments sequentially whereas the parallel segment fetching method requests media segments in parallel. First, a novel rate adaptation metric is presented in this paper, which is the ratio of the expected segment fetch time (ESFT) and the measured segment fetch time to detect network congestion and spare network capacity quickly. ESFT represents the optimum segment fetch time determined by the media segment duration multiplied by the number of parallel HTTP threads to deliver media segments and the remaining duration to fetch the next segment to keep a certain amount of media time in the client buffer. Second, two novel rate adaptation algorithms are proposed for the serial and the parallel segment fetching methods, respectively, based on the proposed rate adaptation metric. The proposed rate adaptation algorithms use a step-wise switch-up and a multi-step switch-down strategy upon detecting the spare networks capacity and congestion with the proposed rate adaptation metric. To provide a good convergence in the representation level for DASH in CDN, a sliding window is used to measure the latest multiple rate adaptation metrics to determine switch-up. To decide switch-down, a rate adaptation metric is used. Each rate adaptation metric represents a reception of a segment/portion of a segment, which can be fetched from the different edge servers in CDN, hence it can be used to estimate the corresponding edge server bandwidth. To avoid buffer overflow due to a slight mismatch in the optimum representation level and bandwidth, an idling method is used to idle a given duration before sending the next segment. In order to solve the fairness between different clients who compete for bandwidth, the prioritized optimum segment fetch time is assigned to the newly joined clients. The proposed rate adaptation method does not require any transport layer information, which is not available at the application layer without cross layer communication. Simulation results show that the proposed rate adaptation algorithms for the serial and the parallel segment fetching methods quickly adapt the media bitrate to match the end-to-end network capacity, provide an advanced convergence and fairness between different clients and also effectively control buffer underflow and overflow for DASH in CDN. The reported simulation results demonstrate that the parallel rate adaptation outperforms the serial DASH rate adaptation algorithm with respect to achievable media bitrates while the serial rate adaptation is superior to the parallel DASH with respect to the convergence and buffer underflow frequency.