Patrick Maupin

A dynamic overproduce-and-choose strategy for the selection of classifier ensembles

The overproduce-and-choose strategy, which is divided into the overproduction and selection phases, has traditionally focused on finding the most accurate subset of classifiers at the selection phase, and using it to predict the class of all the samples in the test data set. It is therefore, a static classifier ensemble selection strategy. In this paper, we propose a dynamic overproduce-and-choose strategy which combines optimization and dynamic selection in a two-level selection phase to allow the selection of the most confident subset of classifiers to label each test sample individually. The optimization level is intended to generate a population of highly accurate candidate classifier ensembles, while the dynamic selection level applies measures of confidence to reveal the candidate ensemble with the highest degree of confidence in the current decision. Experimental results conducted to compare the proposed method to a static overproduce-and-choose strategy and a classical dynamic classifier selection approach demonstrate that our method outperforms both these selection-based methods, and is also more efficient in terms of performance than combining the decisions of all classifiers in the initial pool.

Overfitting cautious selection of classifier ensembles with genetic algorithms

Information fusion research has recently focused on the characteristics of the decision profiles of ensemble members in order to optimize performance. These characteristics are particularly important in the selection of ensemble members. However, even though the control of overfitting is a challenge in machine learning problems, much less work has been devoted to the control of overfitting in selection tasks. The objectives of this paper are: (1) to show that overfitting can be detected at the selection stage; and (2) to present strategies to control overfitting. Decision trees and k nearest neighbors classifiers are used to create homogeneous ensembles, while single- and multi-objective genetic algorithms are employed as search algorithms at the selection stage. In this study, we use bagging and random subspace methods for ensemble generation. The classification error rate and a set of diversity measures are applied as search criteria. We show experimentally that the selection of classifier ensembles conducted by genetic algorithms is prone to overfitting, especially in the multi-objective case. In this study, the partial validation, backwarding and global validation strategies are tailored for classifier ensemble selection problem and compared. This comparison allows us to show that a global validation strategy should be applied to control overfitting in pattern recognition systems involving an ensemble member selection task. Furthermore, this study has helped us to establish that the global validation strategy can be used to measure the relationship between diversity and classification performance when diversity measures are employed as single-objective functions.

Uncertainty in a situation analysis perspective

This paperproposes a discussion on the role of iincerfainty in situation analysis. An overview of the princi- pal typologies of uncertainty foundin the recent literuture is presented. This wide array of uncet-tainty conceptions is a consequence of the intrinsic richness and ambiguity of nat- ural language, but also a consequence of the complexplivs- ical nature of information. Definitions of a liniited number of concepts are proposed in order to better understand the diflerent facets of uncertainty. The benefits sought are: (I) the avoidance of untimely uses of dejniriorls and models of uncertainty. (2) clarifications allowing links with the al- ready well developed logics of knowledge and belief; and (3) guidelines for the selection of the appropriate mathe- matical model to process uncertainty-based information.

Single and Multi-Objective Genetic Algorithms for the Selection of Ensemble of Classifiers

Many recent works have investigated methods to select subsets of classifiers instead of combining all available classifiers. The majority of these works has concluded that the combiner error rate is better than diversity to guide the selection process in order to identify the best performing subset of classifiers. However, the classifier selection process has to take into account three different aspects: complexity, overfitting and performance. These aspects of the selection process have not yet been tackled simultaneously in the literature. The study presented in this paper, deals with these three aspects in a handwritten digit recognition problem. Different search criteria such as diversity, error rate and number of classifiers are applied in single and multi-objective optimization approaches using genetic algorithms. In our experiments, we observed that error rate applied in a single optimization approach was the best objective function to increase performance. The generalized diversity and interrater agreement measures, combined with error rate in pairs of objective functions were the best measures to reduce complexity and keep good performance in a multi-objective optimization approach. Finally, the performance of the solutions found in both, single and multi-objective optimization processes were increased by applying a global validation method to reduce overfitting.

A dynamic model selection strategy for support vector machine classifiers

The Support Vector Machine (SVM) is a very powerful technique for general pattern recognition purposes but its efficiency in practice relies on the optimal selection of hyper-parameters. A naive or ad hoc choice of values for these can lead to poor performance in terms of generalization error and high complexity of the parameterized models obtained in terms of the number of support vectors identified. The task of searching for optimal hyper-parameters with respect to the aforementioned performance measures is the so-called SVM model selection problem. In this paper we propose a strategy to select optimal SVM models in a dynamic fashion in order to address this problem when knowledge about the environment is updated with new observations and previously parameterized models need to be re-evaluated, and in some cases discarded in favor of revised models. This strategy combines the power of swarm intelligence theory with the conventional grid search method in order to progressively identify and sort out potential solutions using dynamically updated training datasets. Experimental results demonstrate that the proposed method outperforms the traditional approaches tested against it, while saving considerable computational time.

Optimizing nearest neighbour in random subspaces using a multi-objective genetic algorithm

In this work, the authors have evaluated almost 20 millions ensembles of classifiers generated by several methods. Trying to optimize those ensembles based on the nearest neighbours and the random subspaces paradigms, we found that the use of a diversity metric called ambiguity had no better positive impact than plain stochastic search.

An empirical study on diversity measures and margin theory for ensembles of classifiers

The main goal of this paper is to investigate the relationship between two theories widely applied to explain the success of classifiers fusion: diversity measures and margin theory. In order to achieve this, we realized an empirical study which evaluates some classical measures related to these two theories with respect to ensembles accuracy. In particular, this study revealed valuable insights on how these two theories can influence each other, and how the application of margin based measures can be useful for the evaluation and selection of ensembles of classifiers with majority voting.

A PSO-based framework for dynamic SVM model selection

Support Vector Machines (SVM) are very powerful classifiers in theory but their efficiency in practice rely on an optimal selection of hyper-parameters. A naïve or ad hoc choice of values for the latter can lead to poor performance in terms of generalization error and high complexity of parameterized models obtained in terms of the number of support vectors identified. This hyper-parameter estimation with respect to the aforementioned performance measures is often called the model selection problem in the SVM research community. In this paper we propose a strategy to select optimal SVM models in a dynamic fashion in order to attend that when knowledge about the environment is updated with new observations and previously parameterized models need to be re-evaluated, and in some cases discarded in favour of revised models. This strategy combines the power of the swarm intelligence theory with the conventional grid-search method in order to progressively identify and sort out potential solutions using dynamically updated training datasets. Experimental results demonstrate that the proposed method outperforms the traditional approaches tested against it while saving considerable computational time.

Pareto analysis for the selection of classifier ensembles

The overproduce-and-choose strategy involves the generation of an initial large pool of candidate classifiers and it is intended to test different candidate ensembles in order to select the best performing solution. The ensembles error rate, ensemble size and diversity measures are the most frequent search criteria employed to guide this selection. By applying the error rate, we may accomplish the main objective in Pattern Recognition and Machine Learning, which is to find high-performance predictors. In terms of ensemble size, the hope is to increase the recognition rate while minimizing the number of classifiers in order to meet both the performance and low ensemble size requirements. Finally, ensembles can be more accurate than individual classifiers only when classifier members present diversity among themselves. In this paper we apply two Pareto front spread quality measures to analyze the relationship between the three main search criteria used in the overproduce-and-choose strategy. Experimental results conducted demonstrate that the combination of ensemble size and diversity does not produce conflicting multi-objective optimization problems. Moreover, we cannot decrease the generalization error rate by combining this pair of search criteria. However, when the error rate is combined with diversity or the ensemble size, we found that these measures are conflicting objective functions and that the performances of the solutions are much higher.

Ambiguity-guided dynamic selection of ensemble of classifiers

Dynamic classifier selection has traditionally focused on selecting the most accurate classifier to predict the class of a particular test pattern. In this paper we propose a new dynamic selection method to select, from a population of ensembles, the most confident ensemble of classifiers to label the test sample. Such a level of confidence is measured by calculating the ambiguity of the ensemble on each test sample. We show theoretically and experimentally that choosing the ensemble of classifiers, from a population of high accurate ensembles, with lowest ambiguity among its members leads to increase the level of confidence of classification, consequently, increasing the generalization performance. Experimental results conducted to compare the proposed method to static selection and DCS-LA, demonstrate that our method outperforms both DCS-LA and static selection strategies when a population of high accurate ensembles is available.