Xueming Qian

PLBP: An effective local binary patterns texture descriptor with pyramid representation

Local binary pattern (LBP) is an effective texture descriptor which has successful applications in texture classification and face recognition. Many extensions are made for conventional LBP descriptors. One of the extensions is dominant local binary patterns which aim at extracting the dominant local structures in texture images. The second extension is representing LBP descriptors in Gabor transform domain (LGBP). The third extension is multi-resolution LBP (MLBP). Another extension is dynamic LBP for video texture extraction. In this paper, we extend the conventional local binary pattern to pyramid transform domain (PLBP). By cascading the LBP information of hierarchical spatial pyramids, PLBP descriptors take texture resolution variations into account. PLBP descriptors show their effectiveness for texture representation. Comprehensive comparisons are made for LBP, MLBP, LGBP, and PLBP. Performances of no sampling, partial sampling and spatial pyramid sampling approaches for the construction of PLBP texture descriptors are compared. The influences of pyramid generation approaches, and pyramid levels to PLBP based image categorization performances are discussed. Compared to the existing multi-resolution LBP descriptors, PLBP is with satisfactory performances and with low computational costs.

Semantic Annotation of High-Resolution Satellite Images via Weakly Supervised Learning

In this paper, we focus on tackling the problem of automatic semantic annotation of high resolution (HR) optical satellite images, which aims to assign one or several predefined semantic concepts to an image according to its content. The main challenges arise from the difficulty of characterizing complex and ambiguous contents of the satellite images and the high human labor cost caused by preparing a large amount of training examples with high-quality pixel-level labels in fully supervised annotation methods. To address these challenges, we propose a unified annotation framework by combining discriminative high-level feature learning and weakly supervised feature transferring. Specifically, an efficient stacked discriminative sparse autoencoder (SDSAE) is first proposed to learn high-level features on an auxiliary satellite image data set for the land-use classification task. Inspired by the motivation that the encoder of the prelearned SDSAE can be regarded as a generic high-level feature extractor for HR optical satellite images, we then transfer the learned high-level features to semantic annotation. To compensate the difference between the auxiliary data set and the annotation data set, the transferred high-level features are further fine-tuned in a weakly supervised scheme by using the tile-level annotated training data. Finally, the fine-tuning process is formulated as an ultimate optimization problem, which can be solved efficiently with our proposed alternate iterative optimization method. Comprehensive experiments on a publicly available land-use classification data set and an annotation data set demonstrate the superiority of our SDSAE-based high-level feature learning method and the effectiveness of our weakly supervised semantic annotation framework compared with state-of-the-art fully supervised annotation methods.

Text detection, localization, and tracking in compressed video

Video text information plays an important role in semantic-based video analysis, indexing and retrieval. Video texts are closely related to the content of a video. Usually, the fundamental steps of text-based video analysis, browsing and retrieval consist of video text detection, localization, tracking, segmentation and recognition. Video sequences are commonly stored in compressed formats where MPEG coding techniques are often adopted. In this paper, a unified framework for text detection, localization, and tracking in compressed videos using the discrete cosines transform (DCT) coefficients is proposed. A coarse to fine text detection method is used to find text blocks in terms of the block DCT texture intensity information. The DCT texture intensity of an 8x8 block of an intra-frame is approximately represented by seven AC coefficients. The candidate text block regions are further verified and refined. The text block region localization and tracking are carried out by virtue of the horizontal and vertical block texture intensity projection profiles. The appearing and disappearing frames of each text line are determined by the text tracking. The final experimental results show the effectiveness of the proposed methods.

GPS Estimation for Places of Interest From Social Users' Uploaded Photos

Social media has become a very popular way for people to share their photos with friends. Because most of the social images are attached with GPS (geo-tags), a photos GPS information can be estimated with the help of the large geo-tagged image set while using a visual searching based approach. This paper proposes an unsupervised image GPS location estimation approach with hierarchical global feature clustering and local feature refinement. It consists of two parts: an offline system and an online system. In the offline system, a hierarchical structure is constructed for a large-scale offline social image set with GPS information. Representative images are selected for each GPS location refined cluster, and an inverted file structure is proposed. In the online system, when given an input image, its GPS information can be estimated by hierarchical global clusters selection and local feature refinement in the online system. Both the computational cost and GPS estimation performance demonstrates the effectiveness of the proposed hierarchical structure and inverted file structure in our approach.

Scalable Mobile Image Retrieval by Exploring Contextual Saliency

Nowadays, it is very convenient to capture photos by a smart phone. As using, the smart phone is a convenient way to share what users experienced anytime and anywhere through social networks, it is very possible that we capture multiple photos to make sure the content is well photographed. In this paper, an effective scalable mobile image retrieval approach is proposed by exploring contextual salient information for the input query image. Our goal is to explore the high-level semantic information of an image by finding the contextual saliency from multiple relevant photos rather than solely using the input image. Thus, the proposed mobile image retrieval approach first determines the relevant photos according to visual similarity, then mines salient features by exploring contextual saliency from multiple relevant images, and finally determines contributions of salient features for scalable retrieval. Compared with the existing mobile-based image retrieval approaches, our approach requires less bandwidth and has better retrieval performance. We can carry out retrieval with <;200-B data, which is <;5% of existing approaches. Most importantly, when the bandwidth is limited, we can rank the transmitted features according to their contributions to retrieval. Experimental results show the effectiveness of the proposed approach.

Social image tagging with diverse semantics.

We have witnessed the popularity of image-sharing websites for sharing personal experiences through photos on the Web. These websites allow users describing the content of their uploaded images with a set of tags. Those user-annotated tags are often noisy and biased. Social image tagging aims at removing noisy tags and suggests new relevant tags. However, most existing tag enrichment approaches predominantly focus on tag relevance and overlook tag diversity problem. How to make the top-ranked tags covering a wide range of semantic is still an opening, yet challenging, issue. In this paper, we propose an approach to retag social images with diverse semantics. Both the relevance of a tag to image as well as its semantic compensations to the already determined tags are fused to determine the final tag list for a given image. Different from existing image tagging approaches, the top-ranked tags are not only highly relevant to the image but also have significant semantic compensations with each other. Experiments show the effectiveness of the proposed approach.

Effective Fades and Flashlight Detection Based on Accumulating Histogram Difference

Scene change detection is a fundamental step in automatic video indexing, browsing and retrieval. Fade in and fade out are two kinds of gradually changing scenes which are difficult to be detected in comparison with the abruptly changing scenes. The salient character of flashlight effect is the luminance change, which is caused by abrupt appearance or disappearance of the illumination source. Performance of shot boundary detection is not satisfactory for the video sequences containing flashlights, if no flashlight discrimination strategy is adopted. In this paper, an effective fades and flashlight detection method is proposed for both the compressed and uncompressed videos, based on the accumulating histogram difference (AHD). This fades detection method is proposed in terms of their mathematical models. AHDs of all the two consecutive frames during fades transitions can be classified into six cases. The flashlight detection method is proposed based on the AHD and the energy variation characters. AHD and energy variation characters for the starting and ending frames of a flashlight have certain regularities, which can also be expressed by cases. Thus the fades and flashlight detection problems are converted into cases matching ones. Experimental results on several test video sequences with different bit rates show the effectiveness of the proposed AHD based fades and flashlight detection method

Recovering Connected Error Region Based on Adaptive Error Concealment Order Determination

Parts of compressed video streams may be lost or corrupted when being transmitted over bandwidth limited networks and wireless communication networks with error-prone channels. Error concealment (EC) techniques are often adopted at the decoder side to improve the quality of the reconstructed video. Under the conditions of a high rate of data packets that arrives at the decoder corrupted, it is likely that the incorrectly decoded macro-blocks (MBs) are concentrated in a connected region, where important spatial reference information is lost. The conventional EC methods usually carry out the block concealment following a lexicographic scan (from top to bottom and from left to right of the image), which would make the methods ineffective for the case that the corrupted blocks are grouped in a connected region. In this paper, a temporal error concealment method, adaptive error concealment order determination (AECOD), is proposed to recover connected corrupted regions. The processing order of an MB in a connected corrupted region is adaptively determined by analyzing the external boundary patterns of the MBs in its neighborhood. The performances, on several video sequences, of the proposed EC scheme have been compared with those obtained by using other error concealment methods reported in the literature. Experimental results show that the AECOD algorithm can improve the recovery performance with respect to the other considered EC methods.

Mining user-contributed photos for personalized product recommendation

With the advent and popularity of social media, users are willing to share their experiences by photos, reviews, blogs, and so on. The social media contents shared by these users reveal potential shopping needs. Product recommender is not limited to just e-commerce sites, it can also be expanded to social media sites. In this paper, we propose a novel hierarchical user interest mining (Huim) approach for personalized products recommendation. The input of our approach consists of user-contributed photos and user generated content (UGC), which include user-annotated photo tags and the comments from others in a social site. The proposed approach consists of four steps. First, we make full use of the visual information and UGC of its photos to mine users interest. Second, we represent user interest by a topic distribution vector, and apply our proposed Huim to enhance interest-related topics. Third, we also represent each product by a topic distribution vector. Then, we measure the relevance of user and product in the topic space and determine the rank of each product for the user. We conduct a series of experiments on Flickr users and the products from Bing Shopping. Experimental results show the effectiveness of the proposed approach.

Object Categorization Using Hierarchical Wavelet Packet Texture Descriptors

Object categorization plays an important role in computer vision, semantic based image content understanding, and image retrieval. Wavelet packet transform provides a very good observation for the images by sub-band filtering. Different objects have distinctive characteristics in the sub-bands of wavelet packets, which should be discriminative for objects classification. In this paper, an object categorization method using hierarchical wavelet packet texture descriptors is proposed. Comparisons between Gabor texture descriptor, pyramid of histograms of orientation gradients (PHOG) and the proposed hierarchical wavelet packet texture descriptors on the widely used OT, Scene-13 and Sport event datasets are also given. Experimental results show that object categorization performances of the proposed texture descriptors are better than that of Gabor texture descriptor and as good as that of PHOG shape descriptor. Object categorization performances of the texture descriptors under various decomposition levels and wavelet bases are discussed. Performances of texture descriptors of global and local images with different partition patterns are also analyzed.