Tongwei Ren

Depth saliency based on anisotropic center-surround difference

Most previous works on saliency detection are dedicated to 2D images. Recently it has been shown that 3D visual information supplies a powerful cue for saliency analysis. In this paper, we propose a novel saliency method that works on depth images based on anisotropic center-surround difference. Instead of depending on absolute depth, we measure the saliency of a point by how much it outstands from surroundings, which takes the global depth structure into consideration. Besides, two common priors based on depth and location are used for refinement. The proposed method works within a complexity of O(N) and the evaluation on a dataset of over 1000 stereo images shows that our method outperforms state-of-the-art.

Image retargeting based on global energy optimization

This paper proposes a novel image retargeting technique based on global energy optimization. Most existing methods enhance the high energy parts of the original image by pre-defined strategies or local optimization based iterations. They can not achieve the global optimal effect in energy retainment. To solve this problem, our approach formulates image retargeting as a global optimization problem on energy. We first calculate the energy map of the original image. Then, we utilize a constrained linear programming to maximize the retained energy in retargeting. Finally, we propose a pixel fusion based method to generate the retargeted image. To make it more feasible in implementation, we further provide two strategies to reduce the time cost of our approach. We demonstrate the proposed approach by comparing with typical image retargeting methods.

Depth-aware salient object detection using anisotropic center-surround difference

Most previous works on salient object detection concentrate on 2D images. In this paper, we propose to explore the power of depth cue for predicting salient regions. Our basic assumption is that a salient object tends to stand out from its surroundings in 3D space. To measure the object-to-surrounding contrast, we propose a novel depth feature which works on a single depth map. Besides, we integrate the 3D spatial prior into our method for saliency refinement. By sparse sampling and representing the image using superpixels, our method works very fast, whose complexity is linear to the image resolution. To segment the salient object, we also develop a saliency based method using adaptive thresholding and GrabCut. The proposed method is evaluated on two large datasets designed for depth-aware salient object detection. The results compared with several state-of-the-art 2D and depth-aware methods show that our method has the most satisfactory overall performance. HighlightsWe proposed a new depth feature for salient region detection.Spatial prior is integrated for saliency refinement.A saliency-based object segmentation method is presented.We built the largest dataset for depth-aware salient object detection evaluation.

Salient object detection for RGB-D image via saliency evolution

Salient object detection aims to detect the most attractive objects in images, which has been widely used as a fundamental of various multimedia applications. In this paper, we propose a novel salient object detection method for RGB-D images based on evolution strategy. Firstly, we independently generate two saliency maps on color channel and depth channel of a given RGB-D image based on its super-pixels representation. Then, we fuse the two saliency maps with refinement to provide an initial saliency map with high precision. Finally, we utilize cellular automata to iteratively propagate saliency on the initial saliency map and generate the final detection result with complete salient objects. The proposed method is evaluated on two public RGB-D datasets, and the experimental results show that our method outperforms the state-of-the-art methods.

OBSIR: Object-based stereo image retrieval

Recent years, the stereo image has become an emerging media in the field of 3D technology, which leads to an urgent demand of stereo image retrieval. In this paper, we attempt to introduce a framework for object-based stereo image retrieval (OBSIR), which retrieves images containing the similar objects to the one captured in the query image by the user. The proposed approach consists of both online and offline procedures. In the offline procedure, we propose a salient object segmentation method making use of both color and depth to extract objects from each image. The extracted objects are then represented by multiple visual feature descriptors. In order to improve the image search efficiently, we construct an approximate nearest neighbor (ANN) index using cluster-based locality sensitive hashing (LSH). In the online stage, the user may supply the query object by selecting a region of interest (ROI) in the query image, or clicking one of the objects recommended by the salient object detector. For the image retrieval evaluation we build a new dataset containing over 10K stereo images. The experiments on this dataset show that the proposed method can effectively recommend the correct object and the final retrieval result is also better than other baseline methods.

Image retargeting using multi-map constrained region warping

Image retargeting aims to adapt images to various screens with small sizes and arbitrary aspect ratios. In this paper, we propose a novel image retargeting approach based on region warping, which emphasizes the image parts with important content while reducing the visual distortion over the whole image. First, the original image is decomposed into homogeneous regions and further represented by curve-edge trapezoid meshes. Then, two kinds of energy maps, importance map and sensitivity map, are calculated by visual attention model and weighted gradient map respectively. With mesh representation and energy map constraints, image retargeting is formulated to a constrained optimization problem of mesh vertexes relocation. Finally, the target image is generated by separately warping the regions based on the deduced optimal solution. The experiments on different images demonstrate the effective and efficiency of our algorithm.

Soft-assigned bag of features for object tracking

Hard assignment-based bag of features (BoF) representation inevitably brings in quantization errors, which may lead to inaccuracy, even failure in object tracking. In this paper, we propose a novel soft-assigned BoF tracking approach, in which soft assignment is utilized to improve the robustness and discrimination of BoF representation. After labeling the tracked target, we first randomly sample the circle patches with adaptive size within and outside the labeled target, extract the local features from the patches, and construct the codebooks by k-means clustering. When tracking in a new frame, we generate the BoF representation of each candidate target, and select the most similar candidate target in the previous tracked result based on BoF representation. To improve tracking performance, we also continuously update the codebooks and refine the tracking results. Experiments show that our approach outperforms the state-of-the-art tracking methods under complex tracking conditions.

Object proposal on RGB-D images via elastic edge boxes

As a fundamental preprocessing of various multimedia applications, object proposal aims to detect the candidate windows possibly containing arbitrary objects in images with two typical strategies, window scoring and grouping. In this paper, we first analyze the feasibility of improving object proposal performance by integrating window scoring and grouping strategies. Then, we propose a novel object proposal method for RGB-D images, named elastic edge boxes. The initial bounding boxes of candidate object regions are efficiently generated by edge boxes, and further adjusted by grouping the super-pixels within elastic range to obtain more accurate candidate windows. To validate the proposed method, we construct the largest RGB-D image data set NJU1800 for object proposal with balanced object number distribution. The experimental results show that our method can effectively and efficiently generate the candidate windows of object regions and it outperforms the state-of-the-art methods considering both accuracy and efficiency.

How important is location information in saliency detection of natural images

Location information, i.e., the position of content in image plane, is considered as an important supplement in saliency detection. The effect of location information is usually evaluated by integrating it with the selected saliency detection methods and measuring the improvement, which is highly influenced by the selection of saliency methods. In this paper, we provide direct and quantitative analysis of the importance of location information for saliency detection in natural images. We firstly analyze the relationship between content location and saliency distribution on four public image datasets, and validate the distribution by simply treating location based Gaussian distribution as saliency map. To further validate the effectiveness of location information, we propose a location based saliency detection approach, which completely initializes saliency maps with location information and propagate saliency among patches based on color similarity, and discuss the robustness of location information’s effect. The experimental results show that location information plays a positive role in saliency detection, and the proposed method can outperform most state-of-the-art saliency detection methods and handle natural images with different object positions and multiple salient objects.

Constrained sampling for image retargeting

In this paper, we present a new approach for retargeting large images to mobile devices with small screens. As the core of image retargeting, information fidelity is adequately considered in terms of reservations of salient regions, edge integrity, and image layout. By taking these aspects as constraints, image retargeting is formulated as a constrained sampling task. Each pixel in image is first represented with a vector encoding the constraints. Then, pixels with the same vector values combine to form blocks, and the original image is thus converted into a graph representation. Thereafter, the sampling ratio of each block is determined with a balanced minimum cost flow algorithm. Final result is generated by an interpolated sampling scheme and direct scaling. Experiments demonstrate the effectiveness of the proposed approach.