Yujin Zhu

Multi-view learning with Universum

The traditional Multi-view Learning (MVL) studies how to process patterns with multiple information sources. In practice, the MVL is proven to have a significant advantage over the Single-view Learning (SVL). But in most real-world cases, there are only single-source patterns to be dealt with and the existing MVL is unable to be directly applied. In order to solve this problem, an alternative MVL technique was developed for the single-source patterns through reshaping the original vector representation of the single-source patterns into multiple matrix representations in our previous work. Doing so can effectively bring an improved classification performance. This paper aims to generalize the previous MVL through taking advantage of the Universum examples which do not belong to either class of the classification problem. The newly-proposed generalization can not only inherit the advantage of the previous MVL, but also get a prior domain knowledge of the whole data distribution. To our knowledge, it introduces the Universum technique into the MVL for the first time. In the implementation, our previous MVL named MultiV-MHKS is selected as the learning paradigm and incorporate MultiV-MHKS with the Universum technique, which forms a more flexible MVL with the Universum called UMultiV-MHKS for short. The subsequent experiments validate that the proposed UMultiV-MHKS can effectively improve the classification performance over both the original MultiV-MHKS and some other state-of-the-art algorithms. Finally, it is demonstrated that the UMultiV-MHKS can get a tighter generalization risk bound in terms of the Rademacher complexity.

Gravitational fixed radius nearest neighbor for imbalanced problem

We use the gravitational scenario into the fixed radius nearest neighbor rule.The proposed GFRNN deals with imbalanced classification problem.GFRNN does not need any manual parameter setting or coordination.Comparison experiments on 40 datasets validate its effectiveness and efficiency. This paper proposes a novel learning model that introduces the calculation of the pairwise gravitation of the selected patterns into the classical fixed radius nearest neighbor method, in order to overcome the drawback of the original nearest neighbor rule when dealing with imbalanced data. The traditional k nearest neighbor rule is considered to lose power on imbalanced datasets because the final decision might be dominated by the patterns from negative classes in spite of the distance measurements. Differently from the existing modified nearest neighbor learning model, the proposed method named GFRNN has a simple structure and thus becomes easy to work. Moreover, all parameters of GFRNN do not need initializing or coordinating during the whole learning procedure. In practice, GFRNN first selects patterns as candidates out of the training set under the fixed radius nearest neighbor rule, and then introduces the metric based on the modified law of gravitation in the physical world to measure the distance between the query pattern and each candidate. Finally, GFRNN makes the decision based on the sum of all the corresponding gravitational forces from the candidates on the query pattern. The experimental comparison validates both the effectiveness and the efficiency of GFRNN on forty imbalanced datasets, comparing to nine typical methods. As a conclusion, the contribution of this paper is constructing a new simple nearest neighbor architecture to deal with imbalanced classification effectively without any manually parameter coordination, and further expanding the family of the nearest neighbor based rules.

McMatMHKS: A direct multi-class matrixized learning machine

Abstract Multi-class classification learning can be implemented by the decomposition to binary classification or the direct techniques. The decomposition technique simplifies s the original learning problem into a set of binary subproblems, separately learns each one, and then combines their results to make a final decision. While the direct technique learns a set of multi-class classifiers by directly optimizing one single objective function. Plenty of empirical results have shown that the two techniques achieve comparable performance. However, both the techniques are mainly designed for vector-pattern samples at present. These traditional vector-pattern-oriented decomposition technique has been extended to a new type of matrix-pattern-oriented classifiers which obtain better learning performance and reduce the learning time–cost by utilizing the original structural information of the input matrix. To our best knowledge, no direct multi-class learning method for matrix pattern has been proposed so far. Therefore, this paper aims to propose a direct multi-class classification technique to compensate such a missing, which is a natural extension of the vector-based direct multi-class classification technique. Simultaneously, the left or right vector acting on matrix pattern in the multi-class matrixized objective function plays a role of a tradeoff parameter to balance the capacity of learning and generalization. Finally, based on the original binary-classifier Matrix-pattern-oriented Modified Ho-Kashyap classifier named MatMHKS, we design a corresponding Direct Multi-class Matrixized Learning Machine named McMatMHKS. It is the first direct multi-class classification technique for matrix patterns. To validate both feasibility and effectiveness of McMatMHKS, we conduct the comparative experiments on some benchmark datasets with two multi-class support vector machines and MatMHKS with the decomposition technique including both one-vs-one and one-vs-all. The results show that like its vector-oriented counterpart, McMatMHKS not only has comparable classification accuracy and AUC value, but also owns lower time complexity when compared with its corresponding decomposition machines.

Regularized Matrix-Pattern-Oriented Classification Machine with Universum

Regularization has the ability to effectively improve the generalization performance, which is due to its control for model complexity via priori knowledge. Matrixized learning as one kind of regularization methods can help improving classification accuracy and reducing computational complexity when dealing with matrix data. This success is attributed to the exploitation of structural knowledge of matrix data. This paper generalizes the matrixized learning through taking the advantage of the Universum data which does not belong to any class of interest in classification problems. The generalized method can not only keep the structural knowledge of the matrix data themselves, but also acquire a priori domain knowledge from the whole data distribution. In implementation, we incorporate the previous matrixized work MatMHKS with the Universum strategy, and develop a novel regularized matrix-pattern-oriented classification machine named UMatMHKS. The subsequential experiments have validated the effectiveness of the proposed UMatMHKS. The results has shown that the proposed UMatMHKS achieved an improvement in classification accuracy of 1.52% over the MatMHKS and 3.20% over the USVM on UCI benchmark datasets. The UMatMHKS also has a shorter average running time of 0.41 s over the 0.71 s from the MatMHKS on UCI datasets. Three main characteristics of UMatMHKS lie in: (1) making full use of the domain knowledge of the whole data distribution as well as inheriting the advantages of the matrixized learning; (2) applying Universum learning into the matrixized learning framework; (3) owning a tighter generalization risk bound.

Matrixized learning machine with modified pairwise constraints

Matrix-pattern-oriented Classifier Design (MatCD) has been demonstrated to be effective in terms of the classification performance since it utilizes two-sided weight vectors to constrain the matrix-based patterns. However, the existing MatCD might not be able to acquire the prior distribution knowledge, such as the relationship between two patterns. Inspired by the Pairwise Constraints (PC) method, i.e., must-links and cannot-links between the patterns, this paper introduces a new regularization term named Rp with a modified PC method into MatCD. The new classifier design strategy is expected to not only learn the structural information of each pattern itself, but also acquire the prior distribution knowledge about each constrained pair with both the discrimination metric from the traditional PC and the spatial distance measure from the heat kernel method. In practice, this paper selects one typical matrixized classifier named MatMHKS as the basic building block and introduces the term Rp into it. The newly-proposed classifier is named MLMMPC and the subsequent experiments validate the effectiveness of it. Two major contributions of this paper can be concluded as (1) improving the existing matrix-pattern-oriented classifier design techniques and (2) modifying the traditional PC method by combining the discrimination metric and the distance measure together. HighlightsA novel matrix-oriented classification algorithm named MLMMPC is proposed.To improve the original matrix learning framework by a new regularization term R p .To combine pairwise constraints and spatial measure together in R p .Implementability is demonstrated on both image- and vector-based datasets.

Globalized and localized matrix-pattern-oriented classification machine

Graphical abstractDisplay Omitted HighlightsA novel classification algorithm named GLMatMHKS is proposed.To capture more structual information through a new regularization term Rgl.To focus on both global and local view of the input matrix sample space.Effectiveness is validated by comparing it with some classic classifiers.The generalization risk bound of it is proved tighter. Inspired by the matrix-based methods used in feature extraction and selection, one matrix-pattern-oriented classification framework has been designed in our previous work and demonstrated to utilize one matrix pattern itself more effectively to improve the classification performance in practice. However, this matrix-based framework neglects the prior structural information of the whole input space that is made up of all the matrix patterns. This paper aims to overcome such flaw through taking advantage of one structure learning method named Alternative Robust Local Embedding (ARLE). As a result, a new regularization term Rgl is designed, expected to simultaneously represent the globality and the locality of the whole data domain, further boosting the existing matrix-based classification method. To our knowledge, it is the first trial to introduce both the globality and the locality of the whole data space into the matrixized classifier design. In order to validate the proposed approach, the designed Rgl is applied into the previous work matrix-pattern-oriented Ho-Kashyap classifier (MatMHKS) to construct a new globalized and localized MatMHKS named GLMatMHKS. The experimental results on a broad range of data validate that GLMatMHKS not only inherits the advantages of the matrixized learning, but also uses the prior structural information more reasonably to guide the classification machine design.

Regularized fisher linear discriminant through two threshold variation strategies for imbalanced problems

Abstract Fisher Linear Discriminant (FLD) has been widely applied to classification tasks due to its simple structure, analytical optimization, and useful criterion. However, when dealing with imbalanced datasets, even though the weight vector of FLD could be trained correctly to preserve the global distribution information of samples, the threshold of FLD might be seriously misled by the extreme proportion of classes. In order to modify the threshold and preserve the weight vector at the same time so as to improve FLD in imbalanced cases, this paper first regularizes the original FLD in a way inspired by the locality preserving projection, and then utilizes two strategies to optimize the threshold: the multi-thresholds selection strategy trains several FLDs with different empirically-defined thresholds, and then selects the optimal threshold out; the threshold-eliminated strategy generates two hyperplanes parallel to the original one built by FLD, and then utilizes a heuristic similarity metric for prediction. It is seen that the former seeks new threshold instead of the old one, while the latter ignores the original threshold. After introducing both strategies into the regularized FLD, two new classifiers are proposed in this paper and abbreviated as RFLD-S1 and RFLD-S2, respectively. Subsequently, the comprehensive comparison experiments on forty-one datasets among nine typical classifiers validate the effectiveness of the proposed methods. Especially, RFLD-S1 performs better than RFLD-S2 and achieves the best on most datasets.

Boundary-Eliminated Pseudoinverse Linear Discriminant for Imbalanced Problems

Existing learning models for classification of imbalanced data sets can be grouped as either boundary-based or nonboundary-based depending on whether a decision hyperplane is used in the learning process. The focus of this paper is a new approach that leverages the advantage of both approaches. Specifically, our new model partitions the input space into three parts by creating two additional boundaries in the training process, and then makes the final decision based on a heuristic measurement between the test sample and a subset of selected training samples. Since the original hyperplane used by the underlying original classifier will be eliminated, the proposed model is named the boundary-eliminated (BE) model. Additionally, the pseudoinverse linear discriminant (PILD) is adopted for the BE model so as to obtain a novel classifier abbreviated as BEPILD. Experiments validate both the effectiveness and the efficiency of BEPILD, compared with 13 state-of-the-art classification methods, based on 31 imbalanced and 7 standard data sets.

Locality sensitive discriminant matrixized learning machine

Abstract Differently from Vector-pattern-oriented Classifier Design (VecCD), Matrix-pattern-oriented Classifier Design (MatCD) is expected to manipulate matrix-oriented patterns directly rather than turning them into a vector, and further demonstrated its effectiveness. However, some prior information, such as the local sensitive discriminant information among matrix-oriented patterns, might be neglected by MatCD. To overcome such flaw, a new regularization term named R LSD is adopted into MatCD by taking advantage of Locality Sensitive Discriminant Analysis (LSDA) in this paper. In detail, the objective function of LSDA is modified and transformed into the regularization term R LSD to explore the local sensitive discriminant information among matrix-oriented patterns. In the implementation, R LSD is collaborated with one typical MatCD, whose name is Matrix-pattern-oriented Ho-Kashyap Classifier (MatMHKS), so as to create a new classifier based on local sensitive discriminant information named LSDMatMHKS for short. Finally, comprehensive experiments are designed to validate the effectiveness of LSDMatMHKS. The major contributions of this paper can be concluded as (1) improving the classification performance and the learning ability of MatCD, (2) introducing local sensitive discriminant information into MatCD and extending the application scenario of LSDA, and (3) validating and analyzing the feasibility and effectiveness of R LSD .

GMFLLM: A general manifold framework unifying three classic models for dimensionality reduction

Abstract As one of the most important preprocess in pattern recognition, the dimensionality reduction is widely applied to the real-world tasks. In practice, there exist three corresponding well-known models, including the Locality Preserving Projection (LPP), the Linear Discriminant Analysis (LDA), and the Maximum Margin Criterion (MMC). Even though several previous works have revealed the partial relationship among the three, there are no further researches. In this paper, from the perspective of LPP, the complete connections among the three models are demonstrated, and then a new framework named GMFLLM is proposed to unify them. Further, since it is possible to utilize the proposed framework as an underlying platform to design more dimensionality reduction variants of LPP, fourteen new variants developed from GMFLLM are approached and investigated in the experiment. Moreover, the best of them, named as the Between-class concerned DLPP/MMC (BDLPP/MMC), is selected to compare with the other seven existing state-of-the-art methods on six image datasets. Results validate the effectiveness of BDLPP/MMC so as to show the generalization of GMFLLM.