Shaoning Pang

Incremental linear discriminant analysis for classification of data streams

This paper presents a constructive method for deriving an updated discriminant eigenspace for classification when bursts of data that contains new classes is being added to an initial discriminant eigenspace in the form of random chunks. Basically, we propose an incremental linear discriminant analysis (ILDA) in its two forms: a sequential ILDA and a Chunk ILDA. In experiments, we have tested ILDA using datasets with a small number of classes and small-dimensional features, as well as datasets with a large number of classes and large-dimensional features. We have compared the proposed ILDA against the traditional batch LDA in terms of discriminability, execution time and memory usage with the increasing volume of data addition. The results show that the proposed ILDA can effectively evolve a discriminant eigenspace over a fast and large data stream, and extract features with superior discriminability in classification, when compared with other methods.

Incremental Learning of Chunk Data for Online Pattern Classification Systems

This paper presents a pattern classification system in which feature extraction and classifier learning are simultaneously carried out not only online but also in one pass where training samples are presented only once. For this purpose, we have extended incremental principal component analysis (IPCA) and some classifier models were effectively combined with it. However, there was a drawback in this approach that training samples must be learned one by one due to the limitation of IPCA. To overcome this problem, we propose another extension of IPCA called chunk IPCA in which a chunk of training samples is processed at a time. In the experiments, we evaluate the classification performance for several large-scale data sets to discuss the scalability of chunk IPCA under one-pass incremental learning environments. The experimental results suggest that chunk IPCA can reduce the training time effectively as compared with IPCA unless the number of input attributes is too large. We study the influence of the size of initial training data and the size of given chunk data on classification accuracy and learning time. We also show that chunk IPCA can obtain major eigenvectors with fairly good approximation.

Face membership authentication using SVM classification tree generated by membership-based LLE data partition

This paper presents a new membership authentication method by face classification using a support vector machine (SVM) classification tree, in which the size of membership group and the members in the membership group can be changed dynamically. Unlike our previous SVM ensemble-based method, which performed only one face classification in the whole feature space, the proposed method employed a divide and conquer strategy that first performs a recursive data partition by membership-based locally linear embedding (LLE) data clustering, then does the SVM classification in each partitioned feature subset. Our experimental results show that the proposed SVM tree not only keeps the good properties that the SVM ensemble method has, such as a good authentication accuracy and the robustness to the change of members, but also has a considerable improvement on the stability under the change of membership group size.

A modified incremental principal component analysis for on-line learning of feature space and classifier

We have proposed a new concept for pattern classification systems in which feature selection and classifier learning are simultaneously carried out on-line. To realize this concept, Incremental Principal Component Analysis (IPCA) and Evolving Clustering Method (ECM) was effectively combined in the previous work. However, in order to construct a desirable feature space, a threshold value to determine the increase of a new feature shoule be properly given in the original IPCA. To alleviate this problem, we can adopt the accumulation ratio as its criterion. However, in incremental situations, the accumulation ratio must be modified every time a new sample is given. Therefore, to use this ratio as a criterion, we also need to develop a one-pass update algorithm for the ratio. In this paper, we propose an improved algorithm of IPCA in which the accumulation ratio as well as the feature space can be updated on-line without all the past samples. To see if correct feature construction is carried out by this new IPCA algorithm, the recognition performance is evaluated for some standard datasets when ECM is adopted as a prototype learning method in Nearest Neighbor classifier.

Classification consistency analysis for bootstrapping gene selection

Consistency modelling for gene selection is a new topic emerging from recent cancer bioinformatics research. The result of operations such as classification, clustering, or gene selection on a training set is often found to be very different from the same operations on a testing set, presenting a serious consistency problem. In practice, the inconsistency of microarray datasets prevents many typical gene selection methods working properly for cancer diagnosis and prognosis. In an attempt to deal with this problem, this paper proposes a new concept of classification consistency and applies it for microarray gene selection problem using a bootstrapping approach, with encouraging results.

Incremental learning for online face recognition

In this paper, a new approach to face recognition is presented in which not only a classifier but also a feature space of input variables is learned incrementally to adapt to incoming training samples. A benefit of this type of incremental learning is that the search for useful features and the learning of an optimal decision boundary are carried out in an online fashion. To implement this idea, an extended version of incremental principal component analysis (IPCA) and resource allocating network with long-term memory (RAN-LTM) are effectively combined. Using IPCA, a feature space is updated by rotating its eigen-axes and increasing the dimensions to adapt to a new training sample. In RAN-LTM, a small number of training samples called memory items are selected and they are utilized for retraining a classifier to realize an excellent incremental ability. To accommodate the classifier to the evolution of the feature space, we present a way to reconstruct the neural classifier without keeping all of the training samples given previously. In the experiments, the proposed incremental learning model is evaluated over a self-compiled face image database. As the result, we verify that the proposed model works well without serious forgetting and the test performance is improved as the learning stages proceed.

Inductive vs transductive inference, global vs local models: SVM, TSVM, and SVMT for gene expression classification problems

This paper compares inductive-, versus transductive modeling, and also global-, versus local models with the use of SVM for gene expression classification problems. SVM are used in their three variants - inductive SVM, transductive SVM (TSVM), and SVM tree (SVMT) - the last two techniques being recently introduced by the authors. The problem of gene expression classification is used for illustration and four benchmark data sets are used to compare the different SVM methods. The TSVM outperforms the inductive SVM models applied on a small to medium variable (gene) set and a small to medium sample set, while SVMT is superior when the problem is defined with a large data set, or - a large set of variables (e.g. 7,000 genes, with little or no variable pre-selection).

Multiple sequence alignment and artificial neural networks for malicious software detection

Malware is currently a major threat to information and computer security, with the volume and growing diversity of its variants causing major problems to traditional security defenses. Software patches and upgrades to anti-viral packages are typically released only after the malwares key characteristics have been identified through infection, by which time it may be too late to protect systems. Sequence analysis is widely used in bioinformatics for revealing the genetic diversity of organisms and annotating gene functions. This paper adopts a new approach to the problem of malware recognition, which is to use multiple sequence alignment techniques from bioinformatics to align variable length computer viral and worm code so that core, invariant regions of the code occupy fixed positions in the alignment patterns. Data mining (ANNs, symbolic rule extraction) can then be used to learn the critical features that help to determine into which class the aligned patterns fall. Experimental results demonstrate the feasibility of our novel approach for identifying malware code through multiple sequence alignment followed by analysis by ANNs and symbolic rule extraction methods.

Encoding and decoding the knowledge of association rules over SVM classification trees

This paper presents a constructive method for association rule extraction, where the knowledge of data is encoded into an SVM classification tree (SVMT), and linguistic association rule is extracted by decoding of the trained SVMT. The method of rule extraction over the SVMT (SVMT-rule), in the spirit of decision-tree rule extraction, achieves rule extraction not only from SVM, but also over the decision-tree structure of SVMT. Thus, the obtained rules from SVMT-rule have the better comprehensibility of decision-tree rule, meanwhile retains the good classification accuracy of SVM. Moreover, profiting from the super generalization ability of SVMT owing to the aggregation of a group of SVMs, the SVMT-rule is capable of performing a very robust classification on such datasets that have seriously, even overwhelmingly, class-imbalanced data distribution. Experiments with a Gaussian synthetic data, seven benchmark cancers diagnosis, and one application of cell-phone fraud detection have highlighted the utility of SVMT and SVMT-rule on comprehensible and effective knowledge discovery, as well as the superior properties of SVMT-rule as compared to a purely support-vector based rule extraction. (A version of SVMT Matlab software is available online at http://kcir.kedri.info)

Dynamic class imbalance learning for incremental LPSVM

Linear Proximal Support Vector Machines (LPSVMs), like decision trees, classic SVM, etc. are originally not equipped to handle drifting data streams that exhibit high and varying degrees of class imbalance. For online classification of data streams with imbalanced class distribution, we propose a dynamic class imbalance learning (DCIL) approach to incremental LPSVM (IncLPSVM) modeling. In doing so, we simplify a computationally non-renewable weighted LPSVM to several core matrices multiplying two simple weight coefficients. When data addition and/or retirement occurs, the proposed DCIL-IncLPSVM(1) accommodates newly presented class imbalance by a simple matrix and coefficient updating, meanwhile ensures no discriminative information lost throughout the learning process. Experiments on benchmark datasets indicate that the proposed DCIL-IncLPSVM outperforms classic IncSVM and IncLPSVM in terms of F-measure and G-mean metrics. Moreover, our application to online face membership authentication shows that the proposed DCIL-IncLPSVM remains effective in the presence of highly dynamic class imbalance, which usually poses serious problems to previous approaches.