Hae Jong Seo

Action Recognition from One Example

We present a novel action recognition method based on space-time locally adaptive regression kernels and the matrix cosine similarity measure. The proposed method uses a single example of an action as a query to find similar matches. It does not require prior knowledge about actions, foreground/background segmentation, or any motion estimation or tracking. Our method is based on the computation of novel space-time descriptors from the query video which measure the likeness of a voxel to its surroundings. Salient features are extracted from said descriptors and compared against analogous features from the target video. This comparison is done using a matrix generalization of the cosine similarity measure. The algorithm yields a scalar resemblance volume, with each voxel indicating the likelihood of similarity between the query video and all cubes in the target video. Using nonparametric significance tests by controlling the false discovery rate, we detect the presence and location of actions similar to the query video. High performance is demonstrated on challenging sets of action data containing fast motions, varied contexts, and complicated background. Further experiments on the Weizmann and KTH data sets demonstrate state-of-the-art performance in action categorization.

Training-Free, Generic Object Detection Using Locally Adaptive Regression Kernels

We present a generic detection/localization algorithm capable of searching for a visual object of interest without training. The proposed method operates using a single example of an object of interest to find similar matches, does not require prior knowledge (learning) about objects being sought, and does not require any preprocessing step or segmentation of a target image. Our method is based on the computation of local regression kernels as descriptors from a query, which measure the likeness of a pixel to its surroundings. Salient features are extracted from said descriptors and compared against analogous features from the target image. This comparison is done using a matrix generalization of the cosine similarity measure. We illustrate optimality properties of the algorithm using a naive-Bayes framework. The algorithm yields a scalar resemblance map, indicating the likelihood of similarity between the query and all patches in the target image. By employing nonparametric significance tests and nonmaxima suppression, we detect the presence and location of objects similar to the given query. The approach is extended to account for large variations in scale and rotation. High performance is demonstrated on several challenging data sets, indicating successful detection of objects in diverse contexts and under different imaging conditions.

Face Verification Using the LARK Representation

We present a novel face representation based on locally adaptive regression kernel (LARK) descriptors. Our LARK descriptor measures a self-similarity based on “signal-induced distance” between a center pixel and surrounding pixels in a local neighborhood. By applying principal component analysis (PCA) and a logistic function to LARK consecutively, we develop a new binary-like face representation which achieves state-of-the-art face verification performance on the challenging benchmark “Labeled Faces in the Wild” (LFW) dataset. In the case where training data are available, we employ one-shot similarity (OSS) based on linear discriminant analysis (LDA). The proposed approach achieves state-of-the-art performance on both the unsupervised setting and the image restrictive training setting (72.23% and 78.90% verification rates), respectively, as a single descriptor representation, with no preprocessing step. As opposed to combined 30 distances which achieve 85.13%, we achieve comparable performance (85.1%) with only 14 distances while significantly reducing computational complexity.

Nonparametric bottom-up saliency detection by self-resemblance

We present a novel bottom-up saliency detection algorithm. Our method computes so-called local regression kernels (i.e., local features) from the given image, which measure the likeness of a pixel to its surroundings. Visual saliency is then computed using the said “self-resemblance” measure. The framework results in a saliency map where each pixel indicates the statistical likelihood of saliency of a feature matrix given its surrounding feature matrices. As a similarity measure, matrix cosine similarity (a generalization of cosine similarity) is employed. State of the art performance is demonstrated on commonly used human eye fixation data [3] and some psychological patterns.

Detection of human actions from a single example

We present an algorithm for detecting human actions based upon a single given video example of such actions. The proposed method is unsupervised, does not require learning, segmentation, or motion estimation. The novel features employed in our method are based on space-time locally adaptive regression kernels. Our method is based on the dense computation of so-called space-time local regression kernels (i.e. local descriptors) from a query video, which measure the likeness of a voxel to its spatio-temporal surroundings. Salient features are then extracted from these descriptors using principal components analysis (PCA). These are efficiently compared against analogous features from the target video using a matrix generalization of the cosine similarity measure. The algorithm yields a scalar resemblance volume; each voxel indicating the like-lihood of similarity between the query video and all cubes in the target video. By employing non-parametric significance tests and non-maxima suppression, we accurately detect the presence and location of actions similar to the given query video. High performance is demonstrated on a challenging set of action data [8] indicating successful detection of multiple complex actions even in the presence of fast motions.

Visual saliency for automatic target detection, boundary detection, and image quality assessment

We present a visual saliency detection method and its applications. The proposed method does not require prior knowledge (learning) or any pre-processing step. Local visual descriptors which measure the likeness of a pixel to its surroundings are computed from an input image. Self-resemblance measured between local features results in a scalar map where each pixel indicates the statistical likelihood of saliency. Promising experimental results are illustrated for three applications: automatic target detection, boundary detection, and image quality assessment.

Robust flash denoising/deblurring by iterative guided filtering

A practical problem addressed recently in computational photography is that of producing a good picture of a poorly lit scene. The consensus approach for solving this problem involves capturing two images and merging them. In particular, using a flash produces one (typically high signal-to-noise ratio [SNR]) image and turning off the flash produces a second (typically low SNR) image. In this article, we present a novel approach for merging two such images. Our method is a generalization of the guided filter approach of He et al., significantly improving its performance. In particular, we analyze the spectral behavior of the guided filter kernel using a matrix formulation, and introduce a novel iterative application of the guided filter. These iterations consist of two parts: a nonlinear anisotropic diffusion of the noisier image, and a nonlinear reaction-diffusion (residual) iteration of the less noisy one. The results of these two processes are combined in an unsupervised manner. We demonstrate that the proposed approach outperforms state-of-the-art methods for both flash/no-flash denoising, and deblurring.

A Comparison of Some State of the Art Image Denoising Methods

We briefly describe and compare some recent advances in image denoising. In particular, we discuss three leading denoising algorithms, and describe their similarities and differences in terms of both structure and performance. Following a summary of each of these methods, several examples with various images corrupted with simulated and real noise of different strengths are presented. With the help of these experiments, we are able to identify the strengths and weaknesses of these state of the art methods, as well as seek the way ahead towards a definitive solution to the long-standing problem of image denoising.

A non-parametric approach to automatic change detection in MRI images of the brain

We present a novel approach to change detection between two brain MRI scans (reference and target.) The proposed method uses a single modality to find subtle changes; and does not require prior knowledge (learning) of the type of changes to be sought. The method is based on the computation of a local kernel from the reference image, which measures the likeness of a pixel to its surroundings. This kernel is then used as a feature and compared against analogous features from the target image. This comparison is made using cosine similarity. The overall algorithm yields a scalar dissimilarity map (DM), indicating the local statistical likelihood of dissimilarity between the reference and target images. DM values exceeding a threshold then identify meaningful and relevant changes. The proposed method is robust to various challenging conditions including unequal signal strength.

Video denoising using higher order optimal space-time adaptation

The optimal spatial adaptation (OSA) method proposed by Boulanger and Kervrann (2006) has proven to be quite effective for spatially adaptive image denoising. This method, in addition to extending the non-local means (NLM) method of A. Buades et al. (2005), employs an iteratively growing window scheme, and a local estimate of the mean square error to very effectively remove noise from images. By adopting an iteratively growing space-time window, the method was recently extended to 3D for video denoising in J. Boulanger et al. (2007). In the present paper, we demonstrate a simple, but effective improvement on the OSA method in both 2- and 3D. We demonstrate that the OSA implicitly relies on a locally constant model of the underlying signal. Thereby, removing this constraint and introducing the possibility of higher order local regression models, we arrive at a relatively simple modification that results in an improvement in performance. While this improvement is observed in both 2D and 3D, we concentrate on demonstrating it in 3D for the application of video denoising.