Jing Huo

Multi-instance dictionary learning for detecting abnormal events in surveillance videos.

In this paper, a novel method termed Multi-Instance Dictionary Learning (MIDL) is presented for detecting abnormal events in crowded video scenes. With respect to multi-instance learning, each event (video clip) in videos is modeled as a bag containing several sub-events (local observations); while each sub-event is regarded as an instance. The MIDL jointly learns a dictionary for sparse representations of sub-events (instances) and multi-instance classifiers for classifying events into normal or abnormal. We further adopt three different multi-instance models, yielding the Max-Pooling-based MIDL (MP-MIDL), Instance-based MIDL (Inst-MIDL) and Bag-based MIDL (Bag-MIDL), for detecting both global and local abnormalities. The MP-MIDL classifies observed events by using bag features extracted via max-pooling over sparse representations. The Inst-MIDL and Bag-MIDL classify observed events by the predicted values of corresponding instances. The proposed MIDL is evaluated and compared with the state-of-the-art methods for abnormal event detection on the UMN (for global abnormalities) and the UCSD (for local abnormalities) datasets and results show that the proposed MP-MIDL and Bag-MIDL achieve either comparable or improved detection performances. The proposed MIDL method is also compared with other multi-instance learning methods on the task and superior results are obtained by the MP-MIDL scheme.

Abnormal event detection via multi-instance dictionary learning

In this paper, we present a method for detecting abnormal events in videos. In the proposed method, we define an event containing several sub-events. Sub-events can be viewed as instances and an event as a bag of instances in the multi-instance learning formulation. Given labeled events but with the labels of sub-events unknown, the proposed method is able to learn a dictionary together with a classification function. The dictionary is capable of generating discriminant sparse codes of sub-events while the classification function is able to classify an event. This method is suited for scenarios where the label of a sub-event is ambiguous, while the label of a set of sub-events is definite and is easy to obtain. Once the sparse codes of sub-events are generated, the classification of an event is carried out according to the result given by the classification function. An efficient optimization procedure of the proposed method is presented. Experiments show that the method is able to detect abnormal events with comparable or improved accuracy compared with other methods.

Heterogeneous Face Recognition by Margin-Based Cross-Modality Metric Learning

Heterogeneous face recognition deals with matching face images from different modalities or sources. The main challenge lies in cross-modal differences and variations and the goal is to make cross-modality separation among subjects. A margin-based cross-modality metric learning (MCM2L) method is proposed to address the problem. A cross-modality metric is defined in a common subspace where samples of two different modalities are mapped and measured. The objective is to learn such metrics that satisfy the following two constraints. The first minimizes pairwise, intrapersonal cross-modality distances. The second forces a margin between subject specific intrapersonal and interpersonal cross-modality distances. This is achieved by defining a hinge loss on triplet-based distance constraints for efficient optimization. It allows the proposed method to focus more on optimizing distances of those subjects whose intrapersonal and interpersonal distances are hard to separate. The proposed method is further extended to a kernelized MCM2L (KMCM2L). Both methods have been evaluated on an ID card face dataset and two other cross-modality benchmark datasets. Various feature extraction methods have also been incorporated in the study, including recent deep learned features. In extensive experiments and comparisons with the state-of-the-art methods, the MCM2L and KMCM2L methods achieved marked improvements in most cases.

Cross-Modal Metric Learning for AUC Optimization

Cross-modal metric learning (CML) deals with learning distance functions for cross-modal data matching. The existing methods mostly focus on minimizing a loss defined on sample pairs. However, the numbers of intraclass and interclass sample pairs can be highly imbalanced in many applications, and this can lead to deteriorating or unsatisfactory performances. The area under the receiver operating characteristic curve (AUC) is a more meaningful performance measure for the imbalanced distribution problem. To tackle the problem as well as to make samples from different modalities directly comparable, a CML method is presented by directly maximizing AUC. The method can be further extended to focus on optimizing partial AUC (pAUC), which is the AUC between two specific false positive rates (FPRs). This is particularly useful in certain applications where only the performances assessed within predefined false positive ranges are critical. The proposed method is formulated as a log-determinant regularized semidefinite optimization problem. For efficient optimization, a minibatch proximal point algorithm is developed. The algorithm is experimentally verified stable with the size of sampled pairs that form a minibatch at each iteration. Several data sets have been used in evaluation, including three cross-modal data sets on face recognition under various scenarios and a single modal data set, the Labeled Faces in the Wild. Results demonstrate the effectiveness of the proposed methods and marked improvements over the existing methods. Specifically, pAUC-optimized CML proves to be more competitive for performance measures such as Rank-1 and verification rate at FPR = 0.1%.

Variation Robust Cross-Modal Metric Learning for Caricature Recognition

In this paper, a variation robust cross-modal metric learning (VR-CM2L) method is proposed for caricature recognition. The goal of caricature recognition is to match a caricature with a photo. This recognition process needs to deal with all kind of variations including different modalities, facial appearance exaggerations, changes in viewpoint, expression, and illumination, etc. All these variations lead to severe misalignment between features of caricatures and photos. To deal with these problems, a specifically designed facial landmark based feature extraction scheme is proposed, where features of caricatures and photos are extracted using different feature extraction steps. At each facial landmark, features of photos are extracted with fixed viewing angle and scale, while features of caricatures are extracted with different scales and viewing angles. To measure the similarity of these features, multiple cross-modal metrics are learned at different facial landmarks in one optimization framework to guarantee global optimum. As the measured features are from two modalities (caricature and photo), cross-modal metric is used to remove modality variations. Pooling at distance level is used during metric optimization to further align the features of caricatures and photos. The introduced pooling step makes the learning method more robust to variations. Experimental results demonstrate the effectiveness of the proposed method on two caricature datasets with various variations.

Joint Multi-field Siamese Recurrent Neural Network for Entity Resolution.

Entity resolution which deals with determining whether two records refer to the same entity has a wide range of applications in both data cleaning and integration. Traditional approaches focus on using string metrics to calculate the matching scores of recorded pairs or employing the machine learning technique with hand-crafted features. However, the effectiveness of these methods largely depends on designing good domain-specific metric methods or extracting discriminative features with rich domain knowledge. Also, traditional learning-based methods usually ignore the discrepancy between citation’s fields. In this paper, to decrease the impact of information gaps between different fields and fully take advantage of semantical and contextual information in each field, we present a novel joint multi-field siamese recurrent architecture. In particular, our method employs word-based Long Short-Term Memory (LSTM) for the fields with the strong relevance between each word and character-based Recurrent Neural Network (RNN) for the fields with the weak relevance between each word, which can exploit each field’s temporal information effectively. Experimental results on three datasets demonstrate that our model can learn discriminative features and outperforms several baseline methods and other RNN-based methods.

OPML: A one-pass closed-form solution for online metric learning

Abstract To achieve a low computational cost when performing online metric learning for large-scale data, we present a one-pass closed-form solution namely OPML in this paper. Typically, the proposed OPML first adopts a one-pass triplet construction strategy, which aims to use only a very small number of triplets to approximate the representation ability of whole original triplets obtained by batch-manner methods. Then, OPML employs a closed-form solution to update the metric for new coming samples, which leads to a low space (i.e., O(d)) and time (i.e., O(d2)) complexity, where d is the feature dimensionality. In addition, an extension of OPML (namely COPML) is further proposed to enhance the robustness when in real case the first several samples come from the same class (i.e., cold start problem). In the experiments, we have systematically evaluated our methods (OPML and COPML) on three typical tasks, including UCI data classification, face verification, and abnormal event detection in videos, which aims to fully evaluate the proposed methods on different sample number, different feature dimensionalities and different feature extraction ways (i.e., hand-crafted and deeply-learned). The results show that OPML and COPML can obtain the promising performance with a very low computational cost. Also, the effectiveness of COPML under the cold start setting is experimentally verified.

A novel ego-centered academic community detection approach via factor graph model

Extracting ego-centered community from large social networks is a practical problem in many real applications. Previous work for ego-centered community detection usually focuses on topological properties of networks, which ignore the actual social attributes between nodes. In this paper, we formalize the ego-centered community detection problem in a unified factor graph model and employ a parameter learning algorithm to estimate the topic-level social influence, the social relationship strength between nodes as well as community structures of networks. Based on the unified model we can obtain more meaningful ego community compared with traditional methods. Experimental results on co-author network demonstrate the effectiveness and efficiency of the proposed approach.