Ashish Shrivastava

Learning from Simulated and Unsupervised Images through Adversarial Training

With recent progress in graphics, it has become more tractable to train models on synthetic images, potentially avoiding the need for expensive annotations. However, learning from synthetic images may not achieve the desired performance due to a gap between synthetic and real image distributions. To reduce this gap, we propose Simulated+Unsupervised (S+U) learning, where the task is to learn a model to improve the realism of a simulators output using unlabeled real data, while preserving the annotation information from the simulator. We develop a method for S+U learning that uses an adversarial network similar to Generative Adversarial Networks (GANs), but with synthetic images as inputs instead of random vectors. We make several key modifications to the standard GAN algorithm to preserve annotations, avoid artifacts, and stabilize training: (i) a self-regularization term, (ii) a local adversarial loss, and (iii) updating the discriminator using a history of refined images. We show that this enables generation of highly realistic images, which we demonstrate both qualitatively and with a user study. We quantitatively evaluate the generated images by training models for gaze estimation and hand pose estimation. We show a significant improvement over using synthetic images, and achieve state-of-the-art results on the MPIIGaze dataset without any labeled real data.

Multiple Kernel Learning for Sparse Representation-Based Classification

In this paper, we propose a multiple kernel learning (MKL) algorithm that is based on the sparse representation-based classification (SRC) method. Taking advantage of the nonlinear kernel SRC in efficiently representing the nonlinearities in the high-dimensional feature space, we propose an MKL method based on the kernel alignment criteria. Our method uses a two step training method to learn the kernel weights and sparse codes. At each iteration, the sparse codes are updated first while fixing the kernel mixing coefficients, and then the kernel mixing coefficients are updated while fixing the sparse codes. These two steps are repeated until a stopping criteria is met. The effectiveness of the proposed method is demonstrated using several publicly available image classification databases and it is shown that this method can perform significantly better than many competitive image classification algorithms.

Segmenting “simple” objects using RGB-D

Segmenting “simple” objects using low-level visual cues is an important capability for a vision system to learn in an unsupervised manner. We define a “simple” object as a compact region enclosed by depth and/or contact boundary in the scene. We propose a segmentation process to extract all the “simple” objects that builds on the fixation-based segmentation framework [1] that segments a region given a point anywhere inside it. In this work, we augment that framework with a fixation strategy to automatically select points inside the “simple” objects and a post-segmentation process to select only the regions corresponding to the “simple” objects in the scene. A novel characteristic of our approach is the incorporation of border ownership, the knowledge about the object side of a boundary pixel. We evaluate the process on a publicly available RGB-D dataset [2] and find that the proposed method successfully extracts 91.4% of all objects in the dataset.

Learning discriminative dictionaries with partially labeled data

While recent techniques for discriminative dictionary learning have demonstrated tremendous success in image analysis applications, their performance is often limited by the amount of labeled data available for training. Even though labeling images is difficult, it is relatively easy to collect unlabeled images either by querying the web or from public datasets. In this paper, we propose a discriminative dictionary learning technique which utilizes both labeled and unlabeled data for learning dictionaries. Extensive evaluation on existing datasets demonstrate that the proposed method performs significantly better than state of the art dictionary learning approaches when unlabeled images are available for training.

Generalized Dictionaries for Multiple Instance Learning

We present a multi-class multiple instance learning (MIL) algorithm using the dictionary learning framework where the data is given in the form of bags. Each bag contains multiple samples, called instances, out of which at least one belongs to the class of the bag. We propose a noisy-OR model and a generalized mean-based optimization framework for learning the dictionaries in the feature space. The proposed method can be viewed as a generalized dictionary learning algorithm since it reduces to a novel discriminative dictionary learning framework when there is only one instance in each bag. Various experiments using popular vision-related MIL datasets as well as the UNBC-McMaster Pain Shoulder Archive database show that the proposed method performs significantly better than the existing methods.

Design of non-linear discriminative dictionaries for image classification

In recent years there has been growing interest in designing dictionaries for image classification. These methods, however, neglect the fact that data of interest often has non-linear structure. Motivated by the fact that this non-linearity can be handled by the kernel trick, we propose learning of dictionaries in the high-dimensional feature space which are simultaneously reconstructive and discriminative. The proposed optimization approach consists of two main stages- coefficient update and dictionary update. We propose a kernel driven simultaneous orthogonal matching pursuit algorithm for the task of sparse coding in the feature space. The dictionary update step is performed using an approximate but efficient KSVD algorithm in feature space. Extensive experiments on image classification demonstrate that the proposed non-linear dictionary learning method is robust and can perform significantly better than many competitive discriminative dictionary learning algorithms.

Non-linear dictionary learning with partially labeled data

While recent techniques for discriminative dictionary learning have demonstrated tremendous success in image analysis applications, their performance is often limited by the amount of labeled data available for training. Even though labeling images is difficult, it is relatively easy to collect unlabeled images either by querying the web or from public datasets. Using the kernel method, we propose a non-linear discriminative dictionary learning technique which utilizes both labeled and unlabeled data for learning dictionaries in the high-dimensional feature space. Furthermore, we show how this method can be extended for ambiguously labeled classification problem where each training sample has multiple labels and only one of them is correct. Extensive evaluation on existing datasets demonstrates that the proposed method performs significantly better than state of the art dictionary learning approaches when unlabeled images are available for training. HighlightsA dictionary learning method that utilizes labeled and unlabeled data is proposed.Using kernel trick, the proposed formulation is extended to the non-linear case.An efficient optimization procedure is proposed for solving this non-linear problem.Each training sample can have multiple labels and only one of them is correct.

Unsupervised domain adaptation using parallel transport on Grassmann manifold

When designing classifiers for classification tasks, one is often confronted with situations where data distributions in the source domain are different from those present in the target domain. This problem of domain adaptation is an important problem that has received a lot of attention in recent years. In this paper, we study the challenging problem of unsupervised domain adaptation, where no labels are available in the target domain. In contrast to earlier works, which assume a single domain shift between the source and target domains, we allow for multiple domain shifts. Towards this, we develop a novel framework based on the parallel transport of union of the source subspaces on the Grassmann manifold. Various recognition experiments show that this way of modeling data with union of subspaces instead of a single subspace improves the recognition performance.

Dictionary-based multiple instance learning

We present a multi-class, multiple instance learning (MIL) algorithm using the dictionary learning framework where the data is given in the form of bags. Each bag contains multiple samples, called instances, out of which at least one belongs to the class of the bag. We propose a noisy-OR model-based optimization framework for learning the dictionaries. Our method can be viewed as a generalized dictionary learning algorithm since it reduces to a novel discriminative dictionary learning framework when there is only one instance in each bag. Various experiments using the popular MIL datasets show that the proposed method performs better than existing methods.

Multiple kernel-based dictionary learning for weakly supervised classification

In this paper, we develop a multiple instance learning (MIL) algorithm using the dictionary learning framework where the labels are given in the form of positive and negative bags, with each bag containing multiple samples. A positive bag is guaranteed to have only one positive class sample while all the samples in a negative bag belong to the negative class. Given positive and negative bags of data, our method learns appropriate feature space to select positive samples from the positive bags as well as optimal dictionaries to represent data in these bags. We apply this method for digit recognition, action recognition, and gender recognition tasks and demonstrate that the proposed method is robust and can perform significantly better than many competitive two class MIL classification algorithms. HighlightsA multiple instance dictionary learning framework to handle weakly supervised data.MKL for well representation and separation of negative and positive bags.We demonstrate the effectiveness our approach on image classification datasets.