Zhaoqiang Xia

A regularized optimization framework for tag completion and image retrieval

Abstract With the fast expansion of social image sharing websites, the tag-based image retrieval (TBIR) becomes important and prevalent for Internet users to search the social images. However, some user-provided tags of social images are too incomplete and ambiguous to facilitate the social image retrieval. In this paper, we propose a regularized optimization framework to complete the missing tags for social images ( tag completion ). Within the regularized optimization framework, the non-negative matrix factorization (NMF) and the holistic visual diversity minimization are used jointly to make the tag-image matrix completed as the relationships of images and tags are represented to a tag-image matrix. The non-negative matrix factorization casts the tag-image matrix into a latent low-rank space and utilizes the semantic relevance of tags to partially complete the insufficient tags. To take the visual content of images into account, the other objective term representing the holistic visual diversity is appended with the NMF to leverage the content-similar images. Moreover, to ensure the proper corrections and sparseness of tag-image matrix, two regularized factors are also included into the optimization framework. Through conducting the experiments on the benchmark image set with the adequate ground truth, we verify the effectiveness of our proposed approach.

Spontaneous micro-expression spotting via geometric deformation modeling

A probabilistic framework is proposed to detect spontaneous micro-expression clips.The geometric deformation captured by ASM model is utilized as features.The features are robust to subtle head movement and illumination variation.The Adaboost algorithm is used to estimate the initial probability for each frame.The random walk algorithm computes the transition probability by deformation similarity.Extensive experiments are performed on two spontaneous datasets. Facial micro-expression is important and prevalent as it reveals the actual emotion of humans. Especially, the automated micro-expression analysis substituted for humans begins to gain the attention recently. However, largely unsolved problems of detecting micro-expressions for subsequent analysis need to be addressed sequentially, such as subtle head movements and unconstrained lighting conditions. To face these challenges, we propose a probabilistic framework to detect spontaneous micro-expression clips temporally from a video sequence (micro-expression spotting) in this paper. In the probabilistic framework, a random walk model is presented to calculate the probability of individual frames having micro-expressions. The Adaboost model is utilized to estimate the initial probability for each frame and the correlation between frames would be considered into the random walk model. The active shape model and Procrustes analysis, which are robust to the head movement and lighting variation, are used to describe the geometric shape of human face. Then the geometric deformation would be modeled and used for Adaboost training. Through performing the experiments on two spontaneous micro-expression datasets, we verify the effectiveness of our proposed micro-expression spotting approach.

An original face anti-spoofing approach using partial convolutional neural network

Recently deep Convolutional Neural Networks have been successfully applied in many computer vision tasks and achieved promising results. So some works have introduced the deep learning into face anti-spoofing. However, most approaches just use the final fully-connected layer to distinguish the real and fake faces. Inspired by the idea of each convolutional kernel can be regarded as a part filter, we extract the deep partial features from the convolutional neural network (CNN) to distinguish the real and fake faces. In our prosed approach, the CNN is fine-tuned firstly on the face spoofing datasets. Then, the block principle component analysis (PCA) method is utilized to reduce the dimensionality of features that can avoid the over-fitting problem. Lastly, the support vector machine (SVM) is employed to distinguish the real the real and fake faces. The experiments evaluated on two public available databases, Replay-Attack and CASIA, show the proposed method can obtain satisfactory results compared to the state-of-the-art methods.

Towards Facial Expression Recognition in the Wild: A New Database and Deep Recognition System

Automatic facial expression recognition (FER) plays an important role in many fields. However, most existing FER techniques are devoted to the tasks in the constrained conditions, which are different from actual emotions. To simulate the spontaneous expression, the number of samples in acted databases is usually small, which limits the ability of facial expression classification. In this paper, a novel database for natural facial expression is constructed leveraging the social images and then a deep model is trained based on the naturalistic dataset. An amount of social labeled images are obtained from the image search engines by using specific keywords. The algorithms of junk image cleansing are then utilized to remove the mislabeled images. Based on the collected images, the deep convolutional neural networks are learned to recognize these spontaneous expressions. Experiments show the advantages of the constructed dataset and deep approach.

Deep convolutional hashing using pairwise multi-label supervision for large-scale visual search

Abstract Image hashing has attracted much attention in the field of large-scale visual search, and learning based approaches have benefited from recent advances of deep learning, which outperforms the shallow models. Most existing deep hashing approaches tend to learn hierarchical models with single-label images limiting the semantic representations. However, few methods have utilized multi-label images to explore rich semantic supervision. In this paper, we propose a new deep convolutional hashing approach by leveraging multi-label images and exploring the label relevance. The proposed method utilizes pairwise supervision to hierarchically transform images into hash codes. Within the deep hashing framework, the Convolutional Neural Networks (CNNs) are considered to automatically learn visual features with smaller semantic gaps. Then a hashing layer using nonlinear mapping is employed to obtain hash codes. A regularized loss function based on pairwise multi-label supervision is proposed to simultaneously learn the features and hash codes. Besides, pairwise multi-label supervision utilizes label relevance to compute semantic similarity of images. The experiments of visual search on two multi-label datasets demonstrate the competitiveness of our proposed approach compared to several state-of-the-art multi-label approaches.

Automatic Abstract Tag Detection for Social Image Tag Refinement and Enrichment

Collaborative image tagging systems, such as Flickr, are very attractive for supporting keyword-based image retrieval, but some user-provided tags of collaboratively-tagged social images might be imprecise. Some people may use general or high-level words (i.e., abstract tags) to tag their images for saving time and effort, but such general or high-level tags are too abstract to describe the visual content of social images precisely. As a result, users may not be able to find what they need when they use the specific keywords for query specification. To tackle the problem of abstract tags, an ontology with three-level semantics is constructed for detecting the candidates of abstract tags from large-scale social images. Then the image context (nearest neighbors) and tag context (most relevant tags) of social images with abstract tags are used to ultimately confirm whether these candidates are abstract or not and identify the specific tags which can further depict the images with abstract tags. In addition, all the relevant tags, which correspond with intermediate nodes between the abstract tags and specific tags on our concept ontology, are added to enrich the tags of social images so that users can have more choices to select various keywords for query specification. We have tested our proposed algorithms on two types of data sets (revised standard datasets and self-constructed dataset) and compared our approach with other approaches.

Kinship Verification from Faces via Similarity Metric Based Convolutional Neural Network

The ability to automatically determine whether two persons are from the same family or not is referred to as Kinship (or family) verification. This is a recent and challenging research topic in computer vision. We propose in this paper a novel approach to kinship verification from facial images. Our solution uses similarity metric based convolutional neural networks. The system is trained using Siamese architecture specific constraints. Extensive experiments on the benchmark KinFaceW-I & II kinship face datasets showed promising results compared to many state-of-the-art methods.

Automatic tag-to-region assignment via multiple instance learning

Translating image tags at the image level to regions (i.e., tag-to-region assignment), which could play an important role in leveraging loosely-labeled training images for object classifier training, has become a popular research topic in the multimedia research community. In this paper, a novel two-stage multiple instance learning algorithm is presented for automatic tag-to-region assignment. The regions are generated by performing multiple-scale image segmentation and the instances with unique semantics are selected out from those regions by a random walk process. The affinity propagation (AP) clustering technique and Hausdorff distance are performed on the instances to identify the most positive instance and utilize it to initialize the maximum searching of Diverse Density likelihood in the first stage. In the second stage, the most contributive instance, which is chosen from each bag, is treated as the key instance for simplifying the computing procedure of Diverse Density likelihood. At last, an automatic method is proposed to discriminate the boundary between positive instances and negative instances. Our experiments on three well-known image sets have provided positive results.

Content-Irrelevant Tag Cleansing via Bi-Layer Clustering and Peer Cooperation

User-provided tags for social images have facilitated many fields, such as social image organization, summarization and retrieval. Since the users utilize their own knowledge and personalized language to describe the visual content of social images, these social tags are too imprecise and ambiguous to exploit the social image tagging. In this paper, we discover the content-similar images (peers) and leverage the relationships among these images (peer cooperation) to handle the problem of content-irrelevant tags. A bi-layer clustering framework for discovering content-similar images is proposed to divide image collection into different groups, and the tags of peers in these groups are cleaned jointly based on tag statistics and relevance. The relevance of tags measured by Google Distance is used to generate the first-layer clustering and then the bi-modality similarity of images is used to perform the second-layer clustering. Based on the bi-layer clustering, we utilize peers in a group to identify their content-irrelevant tags. Finally, an extended Fisher’s criterion is proposed to decide the proper number of content-irrelevant tags. To verify the effectiveness of our proposed technique, we conduct the experiments on the social images of Flickr and the standard benchmark. The comparison experiments show that our proposed algorithm achieves positive results for tag cleansing and image retrieval.

Social tag enrichment via automatic abstract tag refinement

Collaborative image tagging systems, such as Flickr, are very attractive for supporting keyword-based image retrieval, but some social tags of these collaboratively-tagged social images might be imprecise. Some people may use general or high-level words (i.e., abstract tags) to tag their images for saving time and effort, thus such general or high-level tags are too abstract to describe the visual content of social images precisely. As a result, users may not be able to find what they need when they use the specific keywords for query specification. To tackle this problem of abstract tags, a concept ontology is constructed for detecting the abstract tags from large-scale social images. The co-occurrence contexts of social tags and k-NN algorithm with Gaussian Weight are used to find the most specific tags which can signify out the abstract tags. In addition, all the relevant keywords, which are corresponded with intermediate nodes between the high-level concepts (abstract tags) and object classes (most specific tags) on our concept ontology, are added to enrich the lists of social tags, so that users can have more choices to select various keywords for query specification. We have tested our proposed algorithms on two data sets with different images.