Jinye Peng

A regularized optimization framework for tag completion and image retrieval

Abstract With the fast expansion of social image sharing websites, the tag-based image retrieval (TBIR) becomes important and prevalent for Internet users to search the social images. However, some user-provided tags of social images are too incomplete and ambiguous to facilitate the social image retrieval. In this paper, we propose a regularized optimization framework to complete the missing tags for social images ( tag completion ). Within the regularized optimization framework, the non-negative matrix factorization (NMF) and the holistic visual diversity minimization are used jointly to make the tag-image matrix completed as the relationships of images and tags are represented to a tag-image matrix. The non-negative matrix factorization casts the tag-image matrix into a latent low-rank space and utilizes the semantic relevance of tags to partially complete the insufficient tags. To take the visual content of images into account, the other objective term representing the holistic visual diversity is appended with the NMF to leverage the content-similar images. Moreover, to ensure the proper corrections and sparseness of tag-image matrix, two regularized factors are also included into the optimization framework. Through conducting the experiments on the benchmark image set with the adequate ground truth, we verify the effectiveness of our proposed approach.

Spontaneous micro-expression spotting via geometric deformation modeling

A probabilistic framework is proposed to detect spontaneous micro-expression clips.The geometric deformation captured by ASM model is utilized as features.The features are robust to subtle head movement and illumination variation.The Adaboost algorithm is used to estimate the initial probability for each frame.The random walk algorithm computes the transition probability by deformation similarity.Extensive experiments are performed on two spontaneous datasets. Facial micro-expression is important and prevalent as it reveals the actual emotion of humans. Especially, the automated micro-expression analysis substituted for humans begins to gain the attention recently. However, largely unsolved problems of detecting micro-expressions for subsequent analysis need to be addressed sequentially, such as subtle head movements and unconstrained lighting conditions. To face these challenges, we propose a probabilistic framework to detect spontaneous micro-expression clips temporally from a video sequence (micro-expression spotting) in this paper. In the probabilistic framework, a random walk model is presented to calculate the probability of individual frames having micro-expressions. The Adaboost model is utilized to estimate the initial probability for each frame and the correlation between frames would be considered into the random walk model. The active shape model and Procrustes analysis, which are robust to the head movement and lighting variation, are used to describe the geometric shape of human face. Then the geometric deformation would be modeled and used for Adaboost training. Through performing the experiments on two spontaneous micro-expression datasets, we verify the effectiveness of our proposed micro-expression spotting approach.

Extracting Local Binary Patterns from Image Key Points: Application to Automatic Facial Expression Recognition

Facial expression recognition has widely been investigated in the literature. The need of accurate facial alignment has however limited the deployment of facial expression systems in real-world applications. In this paper, a novel feature extraction method is proposed. It is based on extracting local binary patterns (LBP) from image key points. The face region is first segmented into six facial components (left eye, right eye, left eyebrow, right eyebrow, nose, and mouth). Then, local binary patterns are extracted only from the edge points of each facial component. Finally, the local binary pattern features are collected into a histogram and fed to an SVM classifier for facial expression recognition. Compared to the traditional LBP methodology extracting the features from all image pixels, our proposed approach extracts LBP features only from a set of points of face components, yielding in more compact and discriminative representations. Furthermore, our proposed approach does not require face alignment. Extensive experimental analysis on the commonly used JAFFE facial expression benchmark database showed very promising results, outperforming those of the traditional local binary pattern approach.

Automatic Abstract Tag Detection for Social Image Tag Refinement and Enrichment

Collaborative image tagging systems, such as Flickr, are very attractive for supporting keyword-based image retrieval, but some user-provided tags of collaboratively-tagged social images might be imprecise. Some people may use general or high-level words (i.e., abstract tags) to tag their images for saving time and effort, but such general or high-level tags are too abstract to describe the visual content of social images precisely. As a result, users may not be able to find what they need when they use the specific keywords for query specification. To tackle the problem of abstract tags, an ontology with three-level semantics is constructed for detecting the candidates of abstract tags from large-scale social images. Then the image context (nearest neighbors) and tag context (most relevant tags) of social images with abstract tags are used to ultimately confirm whether these candidates are abstract or not and identify the specific tags which can further depict the images with abstract tags. In addition, all the relevant tags, which correspond with intermediate nodes between the abstract tags and specific tags on our concept ontology, are added to enrich the tags of social images so that users can have more choices to select various keywords for query specification. We have tested our proposed algorithms on two types of data sets (revised standard datasets and self-constructed dataset) and compared our approach with other approaches.

Automatic tag-to-region assignment via multiple instance learning

Translating image tags at the image level to regions (i.e., tag-to-region assignment), which could play an important role in leveraging loosely-labeled training images for object classifier training, has become a popular research topic in the multimedia research community. In this paper, a novel two-stage multiple instance learning algorithm is presented for automatic tag-to-region assignment. The regions are generated by performing multiple-scale image segmentation and the instances with unique semantics are selected out from those regions by a random walk process. The affinity propagation (AP) clustering technique and Hausdorff distance are performed on the instances to identify the most positive instance and utilize it to initialize the maximum searching of Diverse Density likelihood in the first stage. In the second stage, the most contributive instance, which is chosen from each bag, is treated as the key instance for simplifying the computing procedure of Diverse Density likelihood. At last, an automatic method is proposed to discriminate the boundary between positive instances and negative instances. Our experiments on three well-known image sets have provided positive results.

Labeling faces with names based on the name semantic network

In this study, we propose a method of labeling faces with names in a large number of news images with captions. Other works explored facial similarities to label faces with names that are sensitive to the intra-person appearance variations, and the captions can offer the cues for the correlations of candidate names. Our method combines textual similarity from image captions with visual similarity of face collections of candidate name to automatically recognize celebrities. It does not require any supervisory inputs. It includes two main steps. Firstly, we build a name semantic network based on textual and visual similarity. Secondly, we apply a name semantic network to label face images with names. We perform experiments on the data set which consists of approximate half a million news images from Yahoo news. The experimental results show that the performance of our method is better than the existing algorithms.

Content-Irrelevant Tag Cleansing via Bi-Layer Clustering and Peer Cooperation

User-provided tags for social images have facilitated many fields, such as social image organization, summarization and retrieval. Since the users utilize their own knowledge and personalized language to describe the visual content of social images, these social tags are too imprecise and ambiguous to exploit the social image tagging. In this paper, we discover the content-similar images (peers) and leverage the relationships among these images (peer cooperation) to handle the problem of content-irrelevant tags. A bi-layer clustering framework for discovering content-similar images is proposed to divide image collection into different groups, and the tags of peers in these groups are cleaned jointly based on tag statistics and relevance. The relevance of tags measured by Google Distance is used to generate the first-layer clustering and then the bi-modality similarity of images is used to perform the second-layer clustering. Based on the bi-layer clustering, we utilize peers in a group to identify their content-irrelevant tags. Finally, an extended Fisher’s criterion is proposed to decide the proper number of content-irrelevant tags. To verify the effectiveness of our proposed technique, we conduct the experiments on the social images of Flickr and the standard benchmark. The comparison experiments show that our proposed algorithm achieves positive results for tag cleansing and image retrieval.

Social tag enrichment via automatic abstract tag refinement

Collaborative image tagging systems, such as Flickr, are very attractive for supporting keyword-based image retrieval, but some social tags of these collaboratively-tagged social images might be imprecise. Some people may use general or high-level words (i.e., abstract tags) to tag their images for saving time and effort, thus such general or high-level tags are too abstract to describe the visual content of social images precisely. As a result, users may not be able to find what they need when they use the specific keywords for query specification. To tackle this problem of abstract tags, a concept ontology is constructed for detecting the abstract tags from large-scale social images. The co-occurrence contexts of social tags and k-NN algorithm with Gaussian Weight are used to find the most specific tags which can signify out the abstract tags. In addition, all the relevant keywords, which are corresponded with intermediate nodes between the high-level concepts (abstract tags) and object classes (most specific tags) on our concept ontology, are added to enrich the lists of social tags, so that users can have more choices to select various keywords for query specification. We have tested our proposed algorithms on two data sets with different images.

Unsupervised deep hashing for large-scale visual search

Learning based hashing plays a pivotal role in large-scale visual search. However, most existing hashing algorithms tend to learn shallow models that do not seek representative binary codes. In this paper, we propose a novel hashing approach based on unsupervised deep learning to hierarchically transform features into hash codes. Within the heterogeneous deep hashing framework, the autoencoder layers with specific constraints are considered to model the nonlinear mapping between features and binary codes. Then, a Restricted Boltzmann Machine (RBM) layer with constraints is utilized to reduce the dimension in the hamming space. The experiments on the problem of visual search demonstrate the competitiveness of our proposed approach compared to the state of the art.

Integrating bilingual search results for automatic junk image filtering

Keyword-based image search engines are now very popular for accessing large amounts of Web images on the Internet. Most existing keyword-based image search engines may return large amounts of junk images (which are irrelevant to the given query word), because the text terms that are loosely associated with the Web images are also used for image indexing. The objective of the proposed work is to effectively filter out the junk images from image search results. Therefore, bilingual image search results for the same keyword-based query are integrated to identify the clusters of the junk images and the clusters of the relevant images. Within relevant image clusters, the results are further refined by removing the duplications under a coarse-to-fine structure. Experiments for a large number of bilingual keyword-based queries (5,000 query words) are simultaneously performed on two keyword-based image search engines (Google Images in English and Baidu Images in Chinese), and our experimental results have shown that integrating bilingual image search results can filter out the junk images effectively.