Sangdoo Yun

Action-Decision Networks for Visual Tracking with Deep Reinforcement Learning

This paper proposes a novel tracker which is controlled by sequentially pursuing actions learned by deep reinforcement learning. In contrast to the existing trackers using deep networks, the proposed tracker is designed to achieve a light computation as well as satisfactory tracking accuracy in both location and scale. The deep network to control actions is pre-trained using various training sequences and fine-tuned during tracking for online adaptation to target and background changes. The pre-training is done by utilizing deep reinforcement learning as well as supervised learning. The use of reinforcement learning enables even partially labeled data to be successfully utilized for semi-supervised learning. Through evaluation of the OTB dataset, the proposed tracker is validated to achieve a competitive performance that is three times faster than state-of-the-art, deep network–based trackers. The fast version of the proposed method, which operates in real-time on GPU, outperforms the state-of-the-art real-time trackers.

Attentional Correlation Filter Network for Adaptive Visual Tracking

We propose a new tracking framework with an attentional mechanism that chooses a subset of the associated correlation filters for increased robustness and computational efficiency. The subset of filters is adaptively selected by a deep attentional network according to the dynamic properties of the tracking target. Our contributions are manifold, and are summarised as follows: (i) Introducing the Attentional Correlation Filter Network which allows adaptive tracking of dynamic targets. (ii) Utilising an attentional network which shifts the attention to the best candidate modules, as well as predicting the estimated accuracy of currently inactive modules. (iii) Enlarging the variety of correlation filters which cover target drift, blurriness, occlusion, scale changes, and flexible aspect ratio. (iv) Validating the robustness and efficiency of the attentional mechanism for visual tracking through a number of experiments. Our method achieves similar performance to non real-time trackers, and state-of-the-art performance amongst real-time trackers.

Visual Path Prediction in Complex Scenes with Crowded Moving Objects

This paper proposes a novel path prediction algorithm for progressing one step further than the existing works focusing on single target path prediction. In this paper, we consider moving dynamics of co-occurring objects for path prediction in a scene that includes crowded moving objects. To solve this problem, we first suggest a two-layered probabilistic model to find major movement patterns and their cooccurrence tendency. By utilizing the unsupervised learning results from the model, we present an algorithm to find the future location of any target object. Through extensive qualitative/quantitative experiments, we show that our algorithm can find a plausible future path in complex scenes with a large number of moving objects.

Multi-task learning with over-sampled time-series representation of a trajectory for traffic motion pattern recognition

This paper proposes an efficient feature sampling and multi-task learning scheme for traffic scene analysis, where all classifiers are trained simultaneously by exploiting the correlations among different motion patterns. We make feature descriptors by high dimensional embedding of the time series data for traffic pattern representation. They preserve detailed spatio-temporal information of the underlying event. Pattern specific details are extracted from raw trajectories and embedded into feature descriptors, which ensures their great discriminability. Training data scarcity problem is tackled through amplification of the patterns hidden in raw trajectory via strategic oversampling and employment of joint feature selection procedure while training the models. Experimental results on 4 surveillance datasets, show great improvement in the motion pattern recognition performance, importance of joint feature selection and fast incremental learning ability of the proposed framework.

PaletteNet: Image Recolorization with Given Color Palette

Image recolorization enhances the visual perception of an image for design and artistic purposes. In this work, we present a deep neural network, referred to as PaletteNet, which recolors an image according to a given target color palette that is useful to express the color concept of an image. PaletteNet takes two inputs: a source image to be recolored and a target palette. PaletteNet is then designed to change the color concept of a source image so that the palette of the output image is close to the target palette. To train PaletteNet, the proposed multi-task loss is composed of Euclidean loss and adversarial loss. The experimental results show that the proposed method outperforms the existing recolorization methods. Human experts with a commercial software take on average 18 minutes to recolor an image, while PaletteNet automatically recolors plausible results in less than a second.

Density-Aware Pedestrian Proposal Networks for Robust People Detection in Crowded Scenes

In this paper, we propose a density-aware pedestrian proposal network (DAPPN) for robust people detection in crowded scenes. Conventional pedestrian detectors and object proposal algorithms easily fail to find people in crowded scenes because of severe occlusions among people. Our method utilizes a crowd density map to resolve the occlusion problem. The proposed network is composed of two networks: the proposal network and the selection network. First, the proposal network predicts the initial pedestrian detection proposals and the crowd density map. After that, the selection network selectively picks the final proposals by considering the initial proposals and the crowd density. To validate the performance of the proposed method, experiments are conducted on crowd-scene datasets: WorldExpo10 and PETS2009. The experimental results show that our method outperforms the conventional method and achieves near real-time speed on a GPU (25 fps).

Category Attentional Search for Fast Object Detection by Mimicking Human Visual Perception

In this paper, we propose a novel selective search method to speed up the object detection via category-based attention scheme. The proposed attentional searching strategy is designed to focus on a small set of selected regions where the object category is expected to exist. The selected regions are estimated by mimicking three properties of the attentional scheme of human visual perception: spotlighting interest regions with low-level saliency (saliency attention), focusing on distinctive features for an object category (feature attention), and estimating potential object position by following human gaze path (gaze attention). Also, the time complexity of each attentional scheme is implemented to be low so that it can hardly affect the computational time. To validate the performance of our method, experiments were conducted on the challenging PASCAL VOC dataset. Experimental results show that our method efficiently generates a small number of candidate boxes for object detection (less than 10ms=image), and the combined object detection system achieves more than 2 times faster performance than the baseline with comparable average precision.

Self-Organizing Cascaded Structure of Deformable Part Models for Fast Object Detection

In this paper, we propose a framework which self-organizes the cascaded object detection filters for fast object detection with maintaining high accuracy. The proposed scheme consists of root and part filter modules, which are cascaded in a self-organizing structure. The pruning of non-object regions in low resolution at the root cascade stage is critical for the object detection speed. At root stage, to prune as many non-object regions as possible, we build a root cascade structure using multiple root models. These models are obtained via bagging procedure for non-linear classification of object and non-object parts in an image. Additional speed-up is achieved by determining proper deployment order of part models. We define a discriminability measure for the part models and suggest a self-organizing scheme to generate an efficient order of part models. The proposed method is evaluated through computational experiments with the PASCAL VOC and INRIA datasets, as a result, our method achieves on average more than 2 times faster performance than the original cascade-DPM, with comparable precision scores.

Unsupervised holistic image generation from key local patches

We introduce a new problem of generating an image based on a small number of key local patches without any geometric prior. In this work, key local patches are defined as informative regions of the target object or scene. This is a challenging problem since it requires generating realistic images and predicting locations of parts at the same time. We construct adversarial networks to tackle this problem. A generator network generates a fake image as well as a mask based on the encoder-decoder framework. On the other hand, a discriminator network aims to detect fake images. The network is trained with three losses to consider spatial, appearance, and adversarial information. The spatial loss determines whether the locations of predicted parts are correct. Input patches are restored in the output image without much modification due to the appearance loss. The adversarial loss ensures output images are realistic. The proposed network is trained without supervisory signals since no labels of key parts are required. Experimental results on seven datasets demonstrate that the proposed algorithm performs favorably on challenging objects and scenes.

Variational Autoencoded Regression: High Dimensional Regression of Visual Data on Complex Manifold

This paper proposes a new high dimensional regression method by merging Gaussian process regression into a variational autoencoder framework. In contrast to other regression methods, the proposed method focuses on the case where output responses are on a complex high dimensional manifold, such as images. Our contributions are summarized as follows: (i) A new regression method estimating high dimensional image responses, which is not handled by existing regression algorithms, is proposed. (ii) The proposed regression method introduces a strategy to learn the latent space as well as the encoder and decoder so that the result of the regressed response in the latent space coincide with the corresponding response in the data space. (iii) The proposed regression is embedded into a generative model, and the whole procedure is developed by the variational autoencoder framework. We demonstrate the robustness and effectiveness of our method through a number of experiments on various visual data regression problems.