Guangling Sun

Saliency Detection for Unconstrained Videos Using Superpixel-Level Graph and Spatiotemporal Propagation

This paper proposes an effective spatiotemporal saliency model for unconstrained videos with complicated motion and complex scenes. First, superpixel-level motion and color histograms as well as global motion histogram are extracted as the features for saliency measurement. Then a superpixel-level graph with the addition of a virtual background node representing the global motion is constructed, and an iterative motion saliency (MS) measurement method that utilizes the shortest path algorithm on the graph is exploited to reasonably generate MS maps. Temporal propagation of saliency in both forward and backward directions is performed using efficient operations on inter-frame similarity matrices to obtain the integrated temporal saliency maps with the better coherence. Finally, spatial propagation of saliency both locally and globally is performed via the use of intra-frame similarity matrices to obtain the spatiotemporal saliency maps with the even better quality. The experimental results on two video data sets with various unconstrained videos demonstrate that the proposed model consistently outperforms the state-of-the-art spatiotemporal saliency models on saliency detection performance.

Improving Saliency Detection Via Multiple Kernel Boosting and Adaptive Fusion

This letter proposes a novel framework to improve the saliency detection performance of an existing saliency model, which is used to generate the initial saliency map. First, a novel regional descriptor consisting of regional self-information, regional variance, and regional contrast on a number of features with local, global, and border context is proposed to describe the segmented regions at multiple scales. Then, regarding saliency computation as a regression problem, a multiple kernel boosting method based on support vector regression (MKB-SVR) is proposed to generate the complementary saliency map. Finally, an adaptive fusion method via learning a quality prediction model for saliency maps is proposed to effectively fuse the initial saliency map with the complementary saliency map and obtain the final saliency map with improvement on saliency detection performance. Experimental results on two public datasets with the state-of-the-art saliency models validate that the proposed method consistently improves the saliency detection performance of various saliency models.

Salient region detection for stereoscopic images

In this paper, we propose an effective saliency model, which combines region-level depth, color and spatial information, to detect salient regions in stereoscopic images. Based on region segmentation results of stereoscopic images, depth contrast, depth weighted color contrast, and spatial compactness of color distribution are measured for each region, and combined to generate the region-level saliency map. Experimental results on a public stereoscopic image dataset with ground truths of salient objects demonstrate that the proposed saliency model outperforms the state-of-the-art saliency models.

Depth-Aware Salient Object Detection and Segmentation via Multiscale Discriminative Saliency Fusion and Bootstrap Learning

This paper proposes a novel depth-aware salient object detection and segmentation framework via multiscale discriminative saliency fusion (MDSF) and bootstrap learning for RGBD images (RGB color images with corresponding Depth maps) and stereoscopic images. By exploiting low-level feature contrasts, mid-level feature weighted factors and high-level location priors, various saliency measures on four classes of features are calculated based on multiscale region segmentation. A random forest regressor is learned to perform the discriminative saliency fusion (DSF) and generate the DSF saliency map at each scale, and DSF saliency maps across multiple scales are combined to produce the MDSF saliency map. Furthermore, we propose an effective bootstrap learning-based salient object segmentation method, which is bootstrapped with samples based on the MDSF saliency map and learns multiple kernel support vector machines. Experimental results on two large datasets show how various categories of features contribute to the saliency detection performance and demonstrate that the proposed framework achieves the better performance on both saliency detection and salient object segmentation.

Saliency detection for RGBD images

Additional depth information from RGBD images is one of characteristics different from conventional 2D images. In this paper, we propose an effective saliency model to detect salient regions in RGBD images. Color contrast and depth contrast are first enhanced with the weighting of depth-based object probability. Then the region merging based saliency refinement is exploited to obtain the color saliency map and depth saliency map, respectively. Finally, a location prior of salient objects is integrated with color saliency and depth saliency to obtain the regional saliency map. Both subjective and objective evaluations on a public RGBD image dataset demonstrate that the proposed saliency model outperforms the state-of-the-art saliency models.

Depth-aware saliency detection using discriminative saliency fusion

In this paper, we propose a multi-stage depth-aware saliency model for salient region detection. We evaluate saliency on different features at low, mid and high levels, by taking account of primary depth and appearance contrasts, different feature weighted factors and location priors, respectively. Unlike most existing depth-aware saliency models that use a linear or experiential fusion formula to combine saliency maps from different features, we calculate saliency of each feature individually at each level and learn a discriminative saliency fusion (DSF) regressor based on random forest to estimate the saliency measures of regions. Both subjective and objective evaluations on two public datasets designed for depth-aware saliency detection demonstrate that the proposed saliency model consistently outperforms the state-of-the-art saliency models on saliency detection performance.

Adaptive saliency fusion based on quality assessment

A variety of saliency models based on different schemes and methods have been proposed in the recent years, and the performance of these models often vary with images and complement each other. Therefore it is a natural idea whether we can elevate saliency detection performance by fusing different saliency models. This paper proposes a novel and general framework to adaptively fuse saliency maps generated using various saliency models based on quality assessment of these saliency maps. Given an input image and its multiple saliency maps, the quality features based on the input image and saliency maps are extracted. Then, a quality assessment model, which is learned using the boosting algorithm with multiple kernels, indicates the quality score of each saliency map. Next, a linear summation method with power-law transformation is exploited to fuse these saliency maps adaptively according to their quality scores. Finally, a graph cut based refinement method is exploited to enhance the spatial coherence of saliency and generate the high-quality final saliency map. Experimental results on three public benchmark datasets with state-of-the-art saliency models demonstrate that our saliency fusion framework consistently outperforms all saliency models and other fusion methods, and effectively elevates saliency detection performance.

Saliency integration driven by similar images

Abstract This paper proposes a saliency integration approach via the use of similar images to elevate saliency detection performance. Given the input image, a group of similar images are first retrieved, and meanwhile, the corresponding multiple saliency maps of the input image are generated by using existing saliency models. Then, the saliency fusion map is generated by using an adaptive fusion method to integrate such saliency maps, for which the fusion weights are measured by the corresponding similarity between each similar image and the input image. Next, an inter-image graph, for each pair of input image and similar image, is constructed to propagate the confident saliency values from the similar image to the input image, yielding the saliency propagation map. Finally, the saliency fusion map and the saliency propagation map are integrated to obtain the final saliency map. Experimental results on two public datasets demonstrate that the proposed approach achieves the better saliency detection performance compared to the existing saliency models and other saliency integration approaches.

Two-Stage Transfer Learning of End-to-End Convolutional Neural Networks for Webpage Saliency Prediction

With the great success of convolutional neural networks (CNN) achieved on various computer vision tasks in recent years, CNN has also been applied in natural image saliency prediction. As a specific visual stimuli, webpages exhibit evident similarities whereas also significant differences from natural image. Consequently, the learned CNN for natural image saliency prediction cannot be directly used to predict webpage saliency. Only a few researches on webpage saliency prediction have been developed till now. In this paper, we propose a simple yet effective scheme of two-stage transfer learning of end-to-end CNN to predict the webpage saliency. In the first stage, the output layer of two typical CNN architectures with instances of AlexNet and VGGNet are reconstructed, and the parameters between the fully connected layers are relearned from a large natural image database for image saliency prediction. In the second stage, the parameters between the fully connected layers are relearned from a scarce webpage database for webpage saliency prediction. In fact, the two-stage transfer learning can be regarded as a task transfer in the first stage and a domain transfer in the second stage, respectively. The experimental results indicate that the proposed two-stage transfer learning of end-to-end CNN can obtain a substantial performance improvement for webpage saliency prediction.

Video saliency detection via bagging-based prediction and spatiotemporal propagation

Abstract The task of spatiotemporal saliency detection is to distinguish the salient objects from background across all the frames in the video. Although many spatiotemporal models have been designed from various aspects, it is still a very challenging task for handing the unconstrained videos with complicated motions and complex scenes. Therefore, in this paper we propose a novel spatiotemporal saliency model to estimate salient objects in unconstrained videos. Specifically, a bagging-based saliency prediction model, i.e. an ensembling regressor, which is the combination of random forest regressors learned from undersampled training sets, is first used to perform saliency prediction for each current frame. Then, both forward and backward propagation within a local temporal window are deployed on each current frame to make a complement to the predicted saliency map and yield the temporal saliency map, in which the backward propagation is constructed based on the temporary saliency estimation of the following frames. Finally, by building the appearance and motion based graphs in a parallel way, spatial propagation is employed over the temporal saliency map to generate the final spatiotemporal saliency map. Through experiments on two challenging datasets, the proposed model consistently outperforms the state-of-the-art models for popping out salient objects in unconstrained videos.