Oscar Divorra Escoda

Depth estimation based on multiview matching with depth/color segmentation and memory efficient Belief Propagation

3D technologies are becoming the more and more relevant in recent years. Visual communications, as well as image and video analysis, benefit in great manner from spatial information such as depth for various applications. Highly accurate visual depth estimation often involves complex optimization algorithms in order to fit proper estimation models to data. From a stereo/multiview matching perspective, local and global algorithms exist. Commonly, the latter are more complex and accurate, as data models are used to take the global structure into account. Belief Propagation has proven to be a good global algorithmic framework for depth estimation. By means of an iterative procedure, it is able to regularize, according to set of local smoothness and geometry constrains, an initial estimation of depth by a local approach such as simple block matching. However, information transfer from iteration to iteration by means of message passing can be excessively demanding in terms of memory bandwidth and usage. In this paper, a new Belief Propagation based algorithm with multiview matching with depth/color segmentation is proposed together with a strategy for message passing compression. Experimental results show the algorithm to be competitive with best performing ones in the state of the art, while reducing by a factor 10 the memory usage, with marginal loss in performance, of a typical Belief Propagation strategy.

A global probabilistic framework for the foreground, background and shadow classification task

Over the years, many works have been published on the two-dimensional foreground segmentation task, describing different methods that treat to extract that part of the scene containing active entities. In most of the cases, the stochastic background process for each pixel is modeled first, and then the foreground pixels are classified as an exception to the model or using maximum a posteriori (MAP) or maximum likelihood (ML). The shadow is usually removed in a later stage and salt and pepper noise is treated with connected component analysis or mathematical morphology. In this paper, we propose a global method that classifies each pixel by finding the best possible class (foreground, background, shadow) examining the image globally. A Markov Random Field is used to represent the dependencies between all the pixels and classes and the global optimal solution is approximated with the Belief Propagation algorithm. The method can extend most local methods and increase their accuracy. In addition, this approach brings a probabilistic justification of the classification problem and it avoids the use of additional post-processing techniques.

Robust foreground segmentation for GPU architecture in an immersive 3D videoconferencing system

Current telepresence systems, while being a great step forward in videoconferencing, still have important points to improve in what eye-contact, gaze and gesture awareness concerns. Many-to-many communications are going to greatly benefit from mature auto-stereoscopic 3D technology; allowing people to engage more natural remote meetings, with proper eye-contact and better spatiality feeling. For this purpose, proper real-time multi-perspective 3D video capture is necessary (often based on one or more View+Depth data sets). Given current state of the art, some sort of foreground segmentation is often necessary at the acquisition in order to generate 3D depth maps with hight enough resolution and accurate object boundaries. For this, one needs flicker-less foreground segmentations, accurate to borders, resilient to noise and foreground shade changes, and able to operate in real-time on performing architectures such as GPGPUs. This paper introduces a robust Foreground Segmentation approach used within the experimental immersive 3D Telepresence system from EU-FP7 3DPresence project. The proposed algorithm is based on a costs minimization using Hierarchical Believe Propagation and outliers reduction by regularization on oversegmented regions. The iterative nature of the approach makes it scalable in complexity, allowing it to increase accuracy and picture size capacity as GPGPUs become faster. In this work, particular care in the design of foreground and background cost models has also been taken in order to overcome limitations of previous work proposed in the literature.

Load and Video Performance Patterns of a Cloud Based WebRTC Architecture

Web Real-Time Communication or Realtime communication in the Web (WebRTC/RTCWeb) is a prolific new standard and technology stack, providing full audio/video agnostic communications for the Web. Service providers implementing such technology deal with various levels of complexity ranging anywhere from: high service distribution, multi-client integration, P2P and Cloud assisted communication backends, content delivery, real-time constraints and across clouds resource allocation. This work presents a study of the joint factors including multi-cloud distribution, network performance, media parameters and back-end resource loads, in Cloud based Media Selective Forwarding Units for WebRTC infrastructures. The monitored workload is sampled from a large population of real users of our testing infrastructure, additionally the performance data is sampled both by passive user measurements as well as server side measurements. Patterns correlating such factors enable designing adaptive resource allocation algorithms and defining media Service Level Objectives (SLO) spanning over multiple data-centers or servers. Based on our analysis, we discover strong periodical load patterns even though the nature of user interaction with the system is mostly not predetermined with variable user churn.

Media streams allocation and load patterns for a WebRTC cloud architecture

Web Real-Time Communication Web Real-Time Communication (WebRTC) is seeing a rapid rise in adoption footprint. This standard provides an audio/video platform-agnostic communications framework for the Web build-in right in the browser. The complex technology stack of a full implementation of the standard is vast and includes elements of various computational disciplines like: content delivery, audio/video processing, media transport and quality of experience control, for both P2P and Cloud relayed communications. To the best of our knowledge, no previous study examines the impact of Cloud back-end load and media quality at production scale for a media stream processing application, as well as load mitigation for Cloud media Selective Forwarding Units. The contribution of this work is the analysis and exploitation of server workload (predictable session size, strong periodical load patterns) and media bit rate patterns that are derived from real user traffic (toward our test environment), over an extended period of time. Additionally, a simple and effective load balancing scheme is discussed to fairly distribute big sessions over multiple servers by exploiting the discovered patterns of stable session sizes and server load predictability. A Cloud simulation environment was built to compare the performance of the algorithm with other load allocation policies. This work is a basis for more advanced resource allocation algorithms and media Service Level Objectives (SLO) spanning multiple Cloud entities.

Prediction matching for video coding

Modern video coding schemes such as H.264/AVC employ multi-hypothesis motion compensation for an improved coding efficiency. However, an additional cost has to be paid for the improved prediction performance in these schemes. Based on the observed high correlation among the multiple hypothesis in H.264/AVC, in this paper, we propose a new method (Prediction Matching) to jointly combine explicit and implicit prediction approaches. The first motion hypothesis on a predicted block is explicitly coded, while the eventual additional hypotheses are implicitly derived at the decoder based on the first one and the available data from previously decoded frames. Thus, the overhead to indicate motion information is reduced, while prediction accuracy may be better with respect to fully implicit multi-hypothesis prediction. Proof-of-concept simulation results show that up to 7.06% bitrate saving with respect to state-of-the-art H.264/AVC can be achieved using our Prediction Matching.