Dmitriy Vatolin

Temporal filtering for depth maps generated by Kinect depth camera

We propose a method of filtering depth maps provided by Kinect depth camera. Filter uses output of the conventional Kinect camera along with the depth sensor to improve the temporal stability of the depth map and fill occlusion areas. To filter input depth map, the algorithm uses the information about motion and color of objects from the video. The proposed method can be applied as a preprocessing stage before using Kinect output data.

Fast video super-resolution via classification

In this paper we propose a novel super-resolution algorithm based on motion compensation and edge-directed spatial interpolation succeeded by fusion via pixel classification. Two high-resolution images are constructed, the first by means of motion compensation and the second by means of edge-directed interpolation. The AdaBoost classifier is then used to fuse these images into an high-resolution frame. Experimental results show that the proposed method surpasses well-known resolution enhancement methods while maintaining moderate computational complexity.

Semiautomatic visual-attention modeling and its application to video compression

This research aims to sufficiently increase the quality of visual-attention modeling to enable practical applications. We found that automatic models are significantly worse at predicting attention than even single-observer eye tracking. We propose a semiautomatic approach that requires eye tracking of only one observer and is based on time consistency of the observers attention. Our comparisons showed the high objective quality of our proposed approach relative to automatic methods and to the results of single-observer eye tracking with no postprocessing. We demonstrated the practical applicability of our proposed concept to the task of saliency-based video compression.

Methodology for stereoscopic motion-picture quality assessment

Creating and processing stereoscopic video imposes additional quality requirements related to view synchronization. In this work we propose a set of algorithms for detecting typical stereoscopic-video problems, which appear owing to imprecise setup of capture equipment or incorrect postprocessing. We developed a methodology for analyzing the quality of S3D motion pictures and for revealing their most problematic scenes. We then processed 10 modern stereo films, including Avatar, Resident Evil: Afterlife and Hugo, and analyzed changes in S3D-film quality over the years. This work presents real examples of common artifacts (color and sharpness mismatch, vertical disparity and excessive horizontal disparity) in the motion pictures we processed, as well as possible solutions for each problem. Our results enable improved quality assessment during the filming and postproduction stages.

Perceptually Motivated Benchmark for Video Matting.

Despite recent progress in the field of video matting, neither public data sets nor even a generally accepted method of measuring quality has yet emerged. In this paper we present an online benchmark for video-matting methods. Using chroma keying and a reflection-aware stop-motion capturing procedure, we prepared 12 test sequences. Then, using subjective data, we performed extensive comparative analysis of different quality metrics. The goal of our benchmark is to enable better understanding of current progress in the field of video matting and to aid in developing new methods.

Automatic detection of artifacts in converted S3D video

In this paper we present algorithms for automatically detecting issues specific to converted S3D content. When a depth-image-based rendering approach produces a stereoscopic image, the quality of the result depends on both the depth maps and the warping algorithms. The most common problem with converted S3D video is edge-sharpness mismatch. This artifact may appear owing to depth-map blurriness at semitransparent edges: after warping, the object boundary becomes sharper in one view and blurrier in the other, yielding binocular rivalry. To detect this problem we estimate the disparity map, extract boundaries with noticeable differences, and analyze edge-sharpness correspondence between views. We pay additional attention to cases involving a complex background and large occlusions. Another problem is detection of scenes that lack depth volume: we present algorithms for detecting at scenes and scenes with at foreground objects. To identify these problems we analyze the features of the RGB image as well as uniform areas in the depth map. Testing of our algorithms involved examining 10 Blu-ray 3D releases with converted S3D content, including Clash of the Titans, The Avengers, and The Chronicles of Narnia: The Voyage of the Dawn Treader. The algorithms we present enable improved automatic quality assessment during the production stage.

Towards automatic stereo-video quality assessment and detection of color and sharpness mismatch

In this paper, we address the problem of stereo-video quality assessment. We introduce objective no-reference metrics for automatic color- and sharpness-mismatch detection in video captured using stereo cameras. The algorithms are based on view matching and reconstruction. A fast block-based color-independent algorithm for stereo matching is proposed. We verify the applicability of the proposed metrics by assessing the quality of full-length films. This quality-assessment procedure reveals scenes distorted during film production or postproduction and enables film comparison in terms of stereoscopic quality.

Automatic left-right channel swap detection

Automatic analysis of stereo-video quality plays an important role in the process of capturing, converting and editing video in 3D format. Although several low-level stereo-video quality metrics have been proposed, mane more-challenging problems of high-level stereo-video analysis, such as left-right channel swap detection, are still practically unresearched. The visual result of a channel swap is very disconcerting, but it is not always obvious even to a human observer what is wrong in such a sequence. In this paper we present a fully automatic algorithm for left-right channel swap detection. Experimental results for real video sequences demonstrate the effectiveness of the proposed technique.

Toward efficient background reconstruction for 3D-view synthesis in dynamic scenes

We propose an efficient method for reconstructing background fragments that are visible in at least one frame of the input video sequence; this method can aid in filling disoccluded areas during view synthesis. We make no assumptions regarding background or camera motion except that it is smooth. The proposed method can therefore handle a wide variety of scenes captured with a free-moving camera. Our approach relies on formulating the problem as frame-by-frame global energy minimization, which we can efficiently solve given that the previous frame provides a good initial approximation. We compare our approach with a commercial background-reconstruction tool for view synthesis and with a general-purpose video-completion algorithm.

A semiautomatic saliency model and its application to video compression

This work aims to apply visual-attention modeling to attention-based video compression. During our comparison we found that eye-tracking data collected even from a single observer outperforms existing automatic models by a significant margin. Therefore, we offer a semiautomatic approach: using computer-vision algorithms and good initial estimation of eye-tracking data from just one observer to produce high-quality saliency maps that are similar to multi-observer eye tracking and that are appropriate for practical applications. We propose a simple algorithm that is based on temporal coherence of the visual-attention distribution and requires eye tracking of just one observer. The results are as good as an average gaze map for two observers. While preparing the saliency-model comparison, we paid special attention to the quality-measurement procedure. We observe that many modern visual-attention models can be improved by applying simple transforms such as brightness adjustment and blending with the center-prior model. The novel quality-evaluation procedure that we propose is invariant to such transforms. To show the practical use of our semiautomatic approach, we developed a saliency-aware modification of the x264 video encoder and performed subjective and objective evaluations. The modified encoder can serve with any attention model and is publicly available.