Yufeng Shan

How many bits does it take for a stimulus to be salient

Visual saliency has been shown to depend on the unpredictability of the visual stimulus given its surround. Various previous works have advocated the equivalence between stimulus saliency and uncompressibility. We propose a direct measure of this quantity, namely the number of bits required by an optimal video compressor to encode a given video patch, and show that features derived from this measure are highly predictive of eye fixations. To account for global saliency effects, these are embedded in a Markov random field model. The resulting saliency measure is shown to achieve state-of-the-art accuracy for the prediction of fixations, at a very low computational cost. Since most modern cameras incorporate video encoders, this paves the way for in-camera saliency estimation, which could be useful in a variety of computer vision applications.

Hybrid video downloading/streaming over peer-to-peer networks

Peer-to-peer based multimedia delivery is becoming increasingly more important in todays networks. Using a peer- to-peer network to assist video streaming is a topic of considerable interest. In this paper, we propose a novel hybrid video downloading/streaming scheme (HDS) that efficiently integrates traditional client/server based video streaming and peer-to-peer based media distribution. Furthermore, we propose a receiver-driven algorithm to coordinate the downloading and streaming modes; and control the state transitions between these modes. We have performed real-world experiments and simulations to validate our concept. These results show that our proposed scheme greatly increases the availability of video content on the receiver side and simultaneously reduces the server load significantly.

Overlay multi-hop FEC scheme for video streaming over peer-to-peer networks

Overlay networks offer promising capabilities for video streaming, due to their support for application-layer processing at the overlay forwarding nodes. In this paper we propose a novel overlay multi-hop FEC (OM-FEC) scheme that provides FEC encoding/decoding capabilities at intermediate nodes in an overlay path. Based on the current network conditions, the end-to-end overlay path is partitioned into segments, and appropriate FEC codes are applied over those segments. We evaluate our work in a real-world scenario and illustrate that the proposed OM-FEC can outperform a pure end-to-end strategy by 10-15 dB in terms of video PSNR.

Two-stage FEC scheme for scalable video transmission over wireless networks

In this paper, we propose a two-stage FEC scheme with an enhanced MAC protocol especially for multimedia data transmission over wireless LANs. The proposed scheme enables the joint optimization of protection strategies across the protocol stack, and packets with errors are delivered to the application layer for correction or drop. In stage 1, packet-level FEC is added across packets at the application layer to correct packet losses due to congestion and route disruption. In stage 2, bit-level FEC is processed within both application packets and stage-one FEC packets to recover from bit errors in the MAC/PHY layer. Header CRC/FEC are used to enhance the MAC/PHY layer and to cooperate with the two stage FEC scheme. Thus, we add FEC only at the application layer, but can correct both application layer packet drops and MAC/PHY layer bit errors. We explore both the efficiency of bandwidth utilization and video performance using the scalable video coder MC-EZBC and ns-2 simulations. Simulation results show that the proposed scheme outperforms conventional IEEE 802.11.

Adaptive two-stage FEC scheme for scalable video transmission over wireless networks

In this paper, we present a two-stage forward error correction (FEC) scheme with an enhanced link-layer protocol especially for multimedia data transmission over wireless LANs. At the application layer, packet-level FEC (stage-one) is added across packets to correct packet losses due to congestion and route disruption. Bit-level FEC (stage-two) is then added to both application packets and stage-one FEC packets to recover bit errors from the link layer. Then at the link layer, header-CRC/FEC is used to enhance protection and to cooperate with the two-stage FEC scheme. The proposed scheme thus provides joint protection across the protocol stack. We explore both its bandwidth efficiency and video performance for the highly efficient and scalable MC-EZBC video codec using the network simulator ns-2. Our results show that the proposed scheme can effectively increase application-layer throughput, reduce both end-to-end transmission delay and application bandwidth fluctuation, and significantly improve video performance.

Compressed-Domain Correlates of Fixations in Video

In this paper we present two compressed-domain features that are highly indicative of saliency in natural video. Their potential to predict saliency is demonstrated by comparing their statistics around human fixation points in a number of videos against the control points selected randomly away from fixations. Using these features, we construct a simple and effective saliency estimation method for compressed video, which utilizes only motion vectors, block coding modes and coded residuals from the bitstream, with partial decoding. The proposed algorithm has been extensively tested on two ground truth datasets using several accuracy metrics. The results indicate its superior performance over several state-of-the-art compressed-domain and pixel-domain algorithms for saliency estimation.

Scalable Video Streaming With Fine-Grain Adaptive Forward Error Correction

In this paper, we investigate a fine-grain adaptive forward error correction (FGA-FEC) coding scheme for scalable video bitstreams. In our work, both the embedded source bitstream and the error-control codes are granularly adapted at block level in intermediate overlay nodes to satisfy heterogeneous users with both different video frame-rate/spatial resolution/quality preferences and different network connections. The proposed FGA-FEC scheme encodes and adapts the embedded source-coded bitstream in such a way that if part of the video source data is actively dropped, parity bits protecting that piece of data are also removed, yielding an efficient result without any transcoding.

Compressed-domain correlates of human fixations in dynamic scenes

In this paper we present two compressed-domain features that are highly indicative of saliency in natural video. We demonstrate the potential of these two features to indicate saliency by comparing their statistics around human fixation points against their statistics at control points away from fixations. Then, using these features, we construct a simple and effective saliency estimation method for compressed video, which utilizes only motion vectors, block coding modes and coded residuals from the bitstream, with partial decoding. The proposed algorithm has been extensively tested on two ground truth datasets using several accuracy metrics. The results indicate its superior performance over several state-of-the-art compressed-domain and pixel-domain algorithms for saliency estimation.

Joint source-network error control coding for scalable overlay video streaming

In this paper, we propose a joint source-network error control coding (JSNC) scheme which efficiently integrates scalable video coding, error control coding and overlay infrastructure to stream video to heterogeneous users. The distributed overlay nodes adapt both the video bitstream and error control coding based on both user requirements and available bandwidth. A novel fine granular adaptive FEC (FGA-FECcheme, a generalization of MD-FEC, is proposed for error recovery during video transmission to heterogeneous users. Encoding once, the FGA-FEC can satisfy multiple heterogeneous users simultaneously without decoding/re-encoding FEC at intermediate nodes.

Comparison of visual saliency models for compressed video

Visual saliency modeling is an increasingly important research problem. While most saliency models for dynamic scenes operate on raw video, several models have also been developed for compressed video. This paper compares the accuracy of nine such models on a common eye-tracking dataset. The results indicate that a reasonably accurate saliency estimation is possible even using only motion vectors from the compressed bitstream. Successful strategies in compressed-domain saliency modeling are highlighted, and certain challenges are identified for future improvement.