Chun-Xia Zhang

RotBoost: A technique for combining Rotation Forest and AdaBoost

This paper presents a novel ensemble classifier generation technique RotBoost, which is constructed by combining Rotation Forest and AdaBoost. The experiments conducted with 36 real-world data sets available from the UCI repository, among which a classification tree is adopted as the base learning algorithm, demonstrate that RotBoost can generate ensemble classifiers with significantly lower prediction error than either Rotation Forest or AdaBoost more often than the reverse. Meanwhile, RotBoost is found to perform much better than Bagging and MultiBoost. Through employing the bias and variance decompositions of error to gain more insight of the considered classification methods, RotBoost is seen to simultaneously reduce the bias and variance terms of a single tree and the decrement achieved by it is much greater than that done by the other ensemble methods, which leads RotBoost to perform best among the considered classification procedures. Furthermore, RotBoost has a potential advantage over AdaBoost of suiting parallel execution.

A local boosting algorithm for solving classification problems

Based on the boosting-by-resampling version of Adaboost, a local boosting algorithm for dealing with classification tasks is proposed in this paper. Its main idea is that in each iteration, a local error is calculated for every training instance and a function of this local error is utilized to update the probability that the instance is selected to be part of next classifiers training set. When classifying a novel instance, the similarity information between it and each training instance is taken into account. Meanwhile, a parameter is introduced into the process of updating the probabilities assigned to training instances so that the algorithm can be more accurate than Adaboost. The experimental results on synthetic and several benchmark real-world data sets available from the UCI repository show that the proposed method improves the prediction accuracy and the robustness to classification noise of Adaboost. Furthermore, the diversity-accuracy patterns of the ensemble classifiers are investigated by kappa-error diagrams.

An empirical study of using Rotation Forest to improve regressors

This paper investigates the performance of Rotation Forest ensemble method in improving the generalization ability of a base predictor for solving regression problems through conducting experiments on several benchmark data sets, which is also compared with that of Bagging, Random Forest, Adaboost.R2, and a single regression tree. The sensitivity of Rotation Forest to the choice of parameters included in it is also studied. On the considered regression data sets, Adaboost.R2 is seen to generally outperform Rotation Forest and both of them are better than Random Forest and a single tree. With respect to Bagging and Rotation Forest, it seems that there is not a clear winner between them. Furthermore, pruning the tree seems to have some bad effect on the performance of all the considered methods.

A sparse-response deep belief network based on rate distortion theory

Abstract Deep belief networks (DBNs) are currently the dominant technique for modeling the architectural depth of brain, and can be trained efficiently in a greedy layer-wise unsupervised learning manner. However, DBNs without a narrow hidden bottleneck typically produce redundant, continuous-valued codes and unstructured weight patterns. Taking inspiration from rate distortion (RD) theory, which encodes original data using as few bits as possible, we introduce in this paper a variant of DBN, referred to as sparse-response DBN (SR-DBN). In this approach, Kullback–Leibler divergence between the distribution of data and the equilibrium distribution defined by the building block of DBN is considered as a distortion function, and the sparse response regularization induced by L 1 -norm of codes is used to achieve a small code rate. Several experiments by extracting features from different scale image datasets show that our approach SR-DBN learns codes with small rate, extracts features at multiple levels of abstraction mimicking computations in the cortical hierarchy, and obtains more discriminative representation than PCA and several basic algorithms of DBNs.

Enhancing Spectral Unmixing by Local Neighborhood Weights

Spectral unmixing is an effective technique to remotely sensed data exploitation. In this paper, appropriate weights in a local neighborhood are designed to enhance spectral unmixing. The weights integrate the spectral and spatial information, and can effectively segment the homogenous and transition areas between different ground cover types. Based on this region-segmentation, pure-pixel-based end-member extraction algorithms are insensitive to the anomalous pixel, and thus perform more robust. In addition, the weights can be used to regularize non-pure-pixel-based unmixing methods, such as nonnegative matrix factorization (NMF). By incorporating the designed local neighborhood weights, a weighted nonnegative matrix factorization (WNMF) algorithm for spectral unmixing is proposed in this paper. Meanwhile, a multiplicative update rule for WNMF is presented, and the monotonic convergence of the rule is proved. Experiments on synthetic and real hyperspectral data validate the effectiveness of the designed weights.

Generalized extreme learning machine autoencoder and a new deep neural network

Extreme learning machine (ELM) is an efficient learning algorithm of training single layer feed-forward neural networks (SLFNs). With the development of unsupervised learning in recent years, integrating ELM with autoencoder has become a new perspective for extracting feature using unlabeled data. In this paper, we propose a new variant of extreme learning machine autoencoder (ELM-AE) called generalized extreme learning machine autoencoder (GELM-AE) which adds the manifold regularization to the objective of ELM-AE. Some experiments carried out on real-world data sets show that GELM-AE outperforms some state-of-the-art unsupervised learning algorithms, including k-means, laplacian embedding (LE), spectral clustering (SC) and ELM-AE. Furthermore, we also propose a new deep neural network called multilayer generalized extreme learning machine autoencoder (ML-GELM) by stacking several GELM-AE to detect more abstract representations. The experiments results show that ML-GELM outperforms ELM and many other deep models, such as multilayer ELM autoencoder (ML-ELM), deep belief network (DBN) and stacked autoencoder (SAE). Due to the utilization of ELM, ML-GELM is also faster than DBN and SAE.

Learning ensemble classifiers via restricted Boltzmann machines

Recently, restricted Boltzmann machines (RBMs) have attracted considerable interest in machine learning field due to their strong ability to extract features. Given some training data, an RBM or a stack of several RBMs can be used to extract informative features. Meanwhile, ensemble learning is an active research area in machine learning owing to their potential to greatly increase the prediction accuracy of a single classifier. However, RBMs have not been studied to work with ensemble learning so far. In this study, we present several methods for integrating RBMs with bagging to generate diverse and accurate individual classifiers. Taking a classification tree as the base learning algorithm, a thoroughly experimental study conducted on 31 real-world data sets yields some promising conclusions. When using the features extracted by RBMs in ensemble learning, the best way is to perform model combination respectively on the original feature set and the one extracted by a single RBM. However, the prediction performance becomes worse when the features detected by a stack of 2 RBMs are also considered. As for the features detected by RBMs, good classification can be obtained only when they are used together with the original features.

A variant of Rotation Forest for constructing ensemble classifiers

Rotation Forest, an effective ensemble classifier generation technique, works by using principal component analysis (PCA) to rotate the original feature axes so that different training sets for learning base classifiers can be formed. This paper presents a variant of Rotation Forest, which can be viewed as a combination of Bagging and Rotation Forest. Bagging is used here to inject more randomness into Rotation Forest in order to increase the diversity among the ensemble membership. The experiments conducted with 33 benchmark classification data sets available from the UCI repository, among which a classification tree is adopted as the base learning algorithm, demonstrate that the proposed method generally produces ensemble classifiers with lower error than Bagging, AdaBoost and Rotation Forest. The bias–variance analysis of error performance shows that the proposed method improves the prediction error of a single classifier by reducing much more variance term than the other considered ensemble procedures. Furthermore, the results computed on the data sets with artificial classification noise indicate that the new method is more robust to noise and kappa-error diagrams are employed to investigate the diversity–accuracy patterns of the ensemble classifiers.

An experimental study of one- and two-level classifier fusion for different sample sizes

Due to the wide variety of fusion techniques available for combining multiple classifiers into a more accurate classifier, a number of good studies have been devoted to determining in what situations some fusion methods should be preferred over other ones. However, the sample size behavior of the various fusion methods has hitherto received little attention in the literature of multiple classifier systems. The main contribution of this paper is thus to investigate the effect of training sample size on their relative performance and to gain more insight into the conditions for the superiority of some combination rules. A large experiment is conducted to study the performance of some fixed and trainable combination rules for executing one- and two-level classifier fusion for different training sample sizes. The experimental results yield the following conclusions: when implementing one-level fusion to combine homogeneous or heterogeneous base classifiers, fixed rules outperform trainable ones in nearly all cases, with only one exception of merging heterogeneous classifiers for large sample size. Moreover, the best classification for any considered sample size is generally achieved by a second level of combination (namely, utilizing one fusion rule to further combine a set of ensemble classifiers with each of them constructed by fusing base classifiers). Under these circumstances, it seems that adopting different types of fusion rules (fixed or trainable) as the combiners for two levels of fusion is appropriate.

A novel method for constructing ensemble classifiers

This paper presents a novel ensemble classifier generation method by integrating the ideas of bootstrap aggregation and Principal Component Analysis (PCA). To create each individual member of an ensemble classifier, PCA is applied to every out-of-bag sample and the computed coefficients of all principal components are stored, and then the principal components calculated on the corresponding bootstrap sample are taken as additional elements of the original feature set. A classifier is trained with the bootstrap sample and some features randomly selected from the new feature set. The final ensemble classifier is constructed by majority voting of the trained base classifiers. The results obtained by empirical experiments and statistical tests demonstrate that the proposed method performs better than or as well as several other ensemble methods on some benchmark data sets publicly available from the UCI repository. Furthermore, the diversity-accuracy patterns of the ensemble classifiers are investigated by kappa-error diagrams.