Mihaela van der Schaar

Fully scalable 3-d overcomplete wavelet video coding using adaptive motion compensated temporal filtering

In this paper, we present a fully scalable 3-D overcomplete wavelet video coder that employs a new and highly efficient 3-D lifting structure for adaptive motion compensated temporal filtering (MCTF). Unlike the conventional interframe wavelet video techniques that apply MCTF on the spatial domain video data and then encode the resulting temporally filtered frames using critical sampled wavelet transforms, the scheme proposed in this paper performs first the spatial domain wavelet transform and subsequently applies MCTF for each wavelet band. To overcome the inefficiency of motion estimation in the wavelet domain, the low band shifting method (LBS) is used at both the encoder and decoder to generate an overcomplete representation of the temporal reference frames. A novel interleaving algorithm for the overcomplete wavelet coefficient is proposed that enables optimal sub-pixel accuracy motion estimation implementations. Furthermore, to achieve arbitrary accuracy motion estimation and compensation in the overcomplete wavelet domain with perfect reconstruction, a novel 3-D lifting structure is also introduced. Simulation results shows that the proposed fully scalable 3-D overcomplete wavelet video coder has comparable or better performance (up to 0.5dB) than the previously proposed interframe wavelet coders under the same coding conditions. Several techniques that can further improve the performance of the proposed overcomplete wavelet coding scheme are also discussed.

Complete-to-overcomplete discrete wavelet transforms for scalable video coding with MCTF

Techniques for full scalability with motion-compensated temporal filtering (MCTF) in the wavelet-domain (in-band) are presented in this paper. The application of MCTF in the wavelet domain is performed after the production of the overcomplete discrete wavelet transform from the critically-sampled decomposition, a process that occurs at both the encoder and decoder side. This process, which is a complete-to-overcomplete discrete wavelet transform, is critical for the efficiency of the system with respect to scalability, coding performance and complexity. We analyze these aspects of the system and set the necessary constraints for drift-free video coding with in-band MCTF. As a result, the proposed architecture permits the independent operation of MCTF within different resolution levels or even different subbands of the transform and allows the successive refinement of the video information in resolution, frame-rate and quality.

Adaptive frequency weighting for fine-granularity-scalability

To compensate for the unpredictability and variability in bandwidth between sender and receiver(s) over the Internet, a new scalable coding tool has recently been introduced in MPEG-4: Fine-Granularity-Scalability (FGS). The FGS framework is very flexible and can adapt in real-time to the Internet bandwidth variations by supporting both SNR and temporal scalability through a single pre-encoded fine-granular stream. However, while FGS is very flexible, it has a lower coding efficiency than non-scalable MPEG-4 coding. To reduce this visual quality penalty at low and medium bit-rates, a subjective quality improvement tool has been introduced in MPEG-4, allowing the prioritized transmission of low frequency DCT coefficients that contribute more to the visual quality. This tool is termed Frequency Weighting (FW). In this paper, we propose a novel scene-characteristic-dependent adaptive FW method that considerably improves the visual quality of FGS at low bit-rates. First, a thorough analysis of the FGS enhancement layer is performed at various bit-rates. Based on this analysis, we concluded that for improved visual quality, different FW methods should be used depending on the video sequence characteristics. Subsequently, we developed an automatic FW matrix adaptation mechanism that measures the brightness, motion and texture activity of each sequence and selects an appropriate FW method for that sequence from a set of a priori determined FW classes. This adaptive FW method has been subjectively evaluated and showed a clear improvement in visual quality compared with non-frequency weighted or non-adaptive frequency weighted sequences.

Fine-granularity-scalability for wireless video and scalable storage

Fine-Granularity-Scalability (FGS) has recently been adopted as the MPEG-4 standard for the efficient and flexible distribution of multimedia over the Internet. The FGS framework allows spatio-temporal-SNR tradeoffs in real-time (not limited to encoding time) at transmission or (post-) processing time, depending upon the available bit-rate, sequence characteristics, priority, user preferences or receiver resources. This paper discusses the versatility of FGS and how it can be used to great advantage in a variety of application domains beyond Internet video streaming, such as robust wireless video transmission and flexible scalable storage. In particular, the focus is on investigating the use of FGS for enabling the Universal Media Access (UMA) paradigm for wireless networks. To cope with detrimental channel conditions, we use several joint source-channel coding schemes for wireless transmission, termed Robust FGS (RFGS). RFGS exploits the FGS property of compressing video data in a set of multi-priority streams, based on the channel characteristics and terminal capabilities, by adapting the channel coding. Our results show that RFGS can provide graceful adaptability to varying network conditions while also offering additional benefits such as flexible Quality-of Service (QoS) management and channel adaptation and allocation. Moreover, the use of FGS for scalable storage of video content in gateways and in-home centralized storage units is also investigated. The results show that the proposed scalable method allows storing ten times or more video data on the same storage device depending on the desired guaranteed minimum quality level.

Differential motion vector coding for scalable coding

Motion compensated wavelet video coding provides very high coding efficiency while enabling spatio-temporal-SNR-complexity scalability. Besides the high degree of adaptabitility, the inherent data prioritization leads to increased robustness in conjunction with unequal error protection (UEP) schemes, and improved error concealment. Hence, motion compensated wavelet video coding schemes are generating great interest for wireless video streaming. Such schemes use motion compensated temporal filtering (MCTF) to remove temporal redundancy. Many extensions to conventional MCTF schemes, that increase the flexibility and the coding efficiency, have been proposed. However these extensions require the coding of additional sets of motion vectors. In this paper, we first define a redundancy factor to identify the additional number of motion vectors that need to be coded with such schemes. We then propose to exploit the temporal correlations between motion vectors to code and estimate them efficiently. We use prediction to reduce the bits needed to code motion vectors. We describe two prediction methods, and highlight the advantages of each scheme. We also use MV prediction during motion estimation, i.e. change the search center and the search range based on the prediction, and describe the tradeoffs to be made between rate, distortion, and complexity. We perform several experiments to illustrate the gains of using temporal prediction, and identify the content dependent nature of results.

Adaptive overcomplete wavelet video coding with spatial transcaling

The unprecedented increase in the level of heterogeneity of emerging wireless networks and the mobile Internet emphasizes the need for scalable and adaptive video solutions both for coding and transmission purposes. However, in general, there is an inherent tradeoff between the level of scalability (e.g., in terms of bitrate range, levels of spatial resolutions, and/or levels of temporal resolutions) and the video-coding penalty incurred by such scalable video schemes as compared to non-scalable coders. In other words, the higher the level of scalability, the lower the overall video quality of the scalable stream that is needed to support the desired scalability level. In [1][2][3], we introduced the notion of TranScaling, which is a generalization of (non-scalable) transcoding. With TranScaling, a scalable video stream covering a given bandwidth range, is mapped into one or more scalable video streams covering different bandwidth ranges. In this paper, we illustrate the benefits of Spatial TranScaling in the context of a recently developed scalable and adaptive inband motion compensated temporal filtering scheme (IBMCTF) [4][5]. We show how using TranScaling, the already high coding efficiency performance of such adaptive IBMCTF schemes can be further improved.

Fine-grained loss protection for robust Internet video streaming

Several embedded video coding schemes have been recently developed for multimedia streaming over IP. In particular, Fine-Granular-Scalability (FGS) video coding has been recently adopted by the MPEG-4 standard as the core video-compression method for streaming applications. From its inception, the FGS scalability structure was designed to be packet-loss resilient especially under Unequal Packet-loss Protection (UPP). In this paper, we introduce the notion of Fine Grained Loss Protection (FGLP), which provides UPP within the FGS enhancement-layer, and we develop an analytical framework for evaluating FGLP bounds. Based on these bounds, we show the impact of applying fine-grained protection to the FGS enhancement-layer for different types of video sequences and over a wide range of bit-rates and packet-loss ratios. Subsequently, the FGLP framework has been implemented using forward error correction codes and its performance for Internet video streaming has been evaluated. As illustrated by our extensive simulation results, fine-grained loss protection for the FGS enhancement- layer can provide significant resilience under moderate-to- high packet-loss ratios (e.g. 5 - 10%).