Mohamed Farouk Abdel Hady

Co-training by Committee: A New Semi-supervised Learning Framework

For many data mining applications, it is necessary to develop algorithms that use unlabeled data to improve the accuracy of the supervised learning. Co-Training is a popular semi-supervised learning algorithm. It assumes that each example is represented by two or more redundantly sufficient sets of features (views) and these views are independent given the class. However, these assumptions are not satisfied in many real-world application domains. Therefore, we present a framework called co-training by committee (CoBC), in which a set of diverse classifiers are used to learn each other. The framework is a simple, general single-view semi-supervised learner that can use any ensemble learner to build diverse committees. Experimental studies on CoBC using bagging, AdaBoost and the random subspace method (RSM) as ensemble learners demonstrate that error diversity among classifiers leads to an effective co-training that requires neither redundant and independent views nor different learning algorithms.

2010 Special Issue: Semi-supervised learning for tree-structured ensembles of RBF networks with Co-Training

Supervised learning requires a large amount of labeled data, but the data labeling process can be expensive and time consuming, as it requires the efforts of human experts. Co-Training is a semi-supervised learning method that can reduce the amount of required labeled data through exploiting the available unlabeled data to improve the classification accuracy. It is assumed that the patterns are represented by two or more redundantly sufficient feature sets (views) and these views are independent given the class. On the other hand, most of the real-world pattern recognition tasks involve a large number of categories which may make the task difficult. The tree-structured approach is an output space decomposition method where a complex multi-class problem is decomposed into a set of binary sub-problems. In this paper, we propose two learning architectures to combine the merits of the tree-structured approach and Co-Training. We show that our architectures are especially useful for classification tasks that involve a large number of classes and a small amount of labeled data where the single-view tree-structured approach does not perform well alone but when combined with Co-Training, it can exploit effectively the independent views and the unlabeled data to improve the recognition accuracy.

Semi-supervised Learning

In traditional supervised learning, one uses ”labeled” data to build a model. However, labeling the training data for real-world applications is difficult, expensive, or time consuming, as it requires the effort of human annotators sometimes with specific domain experience and training. There are implicit costs associated with obtaining these labels from domain experts, such as limited time and financial resources. This is especially true for applications that involve learning with large number of class labels and sometimes with similarities among them. Semi-supervised learning (SSL) addresses this inherent bottleneck by allowing the model to integrate part or all of the available unlabeled data in its supervised learning. The goal is to maximize the learning performance of the model through such newly-labeled examples while minimizing the work required of human annotators. Exploiting unlabeled data to help improve the learning performance has become a hot topic during the last decade and it is divided into four main directions: SSL with graphs, SSL with generative models, semi-supervised support vector machines and SSL by disagreement (SSL with committees). This survey article provides an overview to research advances in this branch of machine learning.

Decision Templates Based RBF Network for Tree-Structured Multiple Classifier Fusion

Multiclass pattern recognition problems (K > 2) can be decomposed by a tree-structured approach. It constructs an ensemble of K -1 individually trained binary classifiers whose predictions are combined to classify unseen instances. A key factor for an effective ensemble is how to combine its member outputs to give the final decision. Although there are various methods to build the tree structure and to solve the underlying binary problems, there is not much work to develop new combination methods that can best combine these intermediate results. We present here a trainable fusion method that integrates statistical information about the individual outputs (clustered decision templates) into a Radial Basis Function (RBF) network. We compare our model with the decision templates combiner and the existing nontrainable tree ensemble fusion methods: classical decision tree-like approach, product of the unique path and Dempster-Shafer evidence theory based method.

Semi-supervised Learning of Tree-Structured RBF Networks Using Co-training

Supervised learning requires a large amount of labeled data but the data labeling process can be expensive and time consuming. Co-training is a semi-supervised learning method that reduces the amount of required labeled data through exploiting the available unlabeled data in supervised learning to boost the accuracy. It assumes that the patterns are described by multiple independent feature sets and each feature set is sufficient for classification. On the other hand, most of the real-world pattern recognition tasks involve a large number of categories which may make the task difficult. The tree-structured approach is a multi-class decomposition strategy where a complex multi-class problem is decomposed into tree structured binary sub-problems. In this paper, we propose a framework that combines the tree-structured approach with Co-training. We show that our framework is especially useful for classification tasks involving a large number of classes and a small amount of labeled data where the tree-structured approach does not perform well by itself and when combined with Co-training, the unlabeled data boosts its accuracy.

Semi-supervised Learning for Regression with Co-training by Committee

Semi-supervised learning is a paradigm that exploits the unlabeled data in addition to the labeled data to improve the generalization error of a supervised learning algorithm. Although in real-world applications regression is as important as classification, most of the research in semi-supervised learning concentrates on classification. In particular, although Co-Training is a popular semi-supervised learning algorithm, there is not much work to develop new Co-Training style algorithms for semi-supervised regression. In this paper, a semi-supervised regression framework, denoted by CoBCReg is proposed, in which an ensemble of diverse regressors is used for semi-supervised learning that requires neither redundant independent views nor different base learning algorithms. Experimental results show that CoBCReg can effectively exploit unlabeled data to improve the regression estimates.

A Multi-objective Genetic Algorithm for Pruning Support Vector Machines

Support vector machines (SVMs) often contain a large number of support vectors which reduce the run-time speeds of decision functions. In addition, this might cause an over fitting effect where the resulting SVM adapts itself to the noise in the training set rather than the true underlying data distribution and will probably fail to correctly classify unseen examples. To obtain more fast and accurate SVMs, many methods have been proposed to prune SVs in trained SVMs. In this paper, we propose a multi-objective genetic algorithm to reduce the complexity of support vector machines as well as to improve generalization accuracy by the reduction of over fitting. Experiments on four benchmark datasets show that the proposed evolutionary approach can effectively reduce the number of support vectors included in the decision functions of SVMs without sacrificing their classification accuracy.

Semi-supervised facial expressions annotation using co-training with fast probabilistic tri-class SVMs

Supervised learning requires a large amount of labeled data but the data labeling process can be expensive and time consuming, as it requires the efforts of human experts. Semi-supervised learning methods that can reduce the amount of required labeled data through exploiting the available unlabeled data to improve the classification accuracy. Here, we propose a learning framework to exploit the unlabeled data by decomposing multi-class problems into a set of binary problems and apply Co-Training to each binary problem. A probabilistic version of Tri-Class Support Vector Machine is proposed (SVM) that can discriminate between ignorance and uncertainty and an updated version of Sequential Minimal Optimization (SMO) algorithm is used for fast learning of Tri-Class SVMs. The proposed framework is applied to facial expressions recognition task. The results show that Co-Training can exploit effectively the independent views and the unlabeled data to improve the recognition accuracy of facial expressions.

Combining committee-based semi-supervised and active learning and its application to handwritten digits recognition

Semi-supervised learning reduces the cost of labeling the training data of a supervised learning algorithm through using unlabeled data together with labeled data to improve the performance. Co-Training is a popular semi-supervised learning algorithm, that requires multiple redundant and independent sets of features (views). In many real-world application domains, this requirement can not be satisfied. In this paper, a single-view variant of Co-Training, CoBC (Co-Training by Committee), is proposed, which requires an ensemble of diverse classifiers instead of the redundant and independent views. Then we introduce two new learning algorithms, QBC-then-CoBC and QBC-with-CoBC, which combines the merits of committee-based semi-supervised learning and committee-based active learning. An empirical study on handwritten digit recognition is conducted where the random subspace method (RSM) is used to create ensembles of diverse C4.5 decision trees. Experiments show that these two combinations outperform the other non committee-based ones.

When classifier selection meets information theory: A unifying view

Classifier selection aims to reduce the size of an ensemble of classifiers in order to improve its efficiency and classification accuracy. Recently an information-theoretic view was presented for feature selection. It derives a space of possible selection criteria and show that several feature selection criteria in the literature are points within this continuous space. The contribution of this paper is to export this information-theoretic view to solve an open issue in ensemble learning which is classifier selection. We investigated a couple of information-theoretic selection criteria that are used to rank classifiers.