Petr Somol

Adaptive floating search methods in feature selection

Abstract A new suboptimal search strategy for feature selection is presented. It represents a more sophisticated version of “classical” floating search algorithms ( Pudil et al., 1994 ), attempts to remove some of their potential deficiencies and facilitates finding a solution even closer to the optimal one.

Fast branch & bound algorithms for optimal feature selection

A novel search principle for optimal feature subset selection using the branch & bound method is introduced. Thanks to a simple mechanism for predicting criterion values, a considerable amount of time can be saved by avoiding many slow criterion evaluations. We propose two implementations of the proposed prediction mechanism that are suitable for use with nonrecursive and recursive criterion forms, respectively. Both algorithms find the optimum usually several times faster than any other known branch & bound algorithm. As the algorithm computational efficiency is crucial, due to the exponential nature of the search problem, we also investigate other factors that affect the search performance of all branch & bound algorithms. Using a set of synthetic criteria, we show that the speed of the branch & bound algorithms strongly depends on the diversity among features, feature stability with respect to different subsets, and criterion function dependence on feature set size. We identify the scenarios where the search is accelerated the most dramatically (finish in linear time), as well as the worst conditions. We verify our conclusions experimentally on three real data sets using traditional probabilistic distance criteria.

Road sign classification using Laplace kernel classifier

Abstract Driver support systems (DSS) of intelligent vehicles will predict potentially dangerous situations in heavy traffic, help with navigation and vehicle guidance and interact with a human driver. Important information necessary for traffic situation understanding is presented by road signs. A new kernel rule has been developed for road sign classification using the Laplace probability density. Smoothing parameters of the Laplace kernel are optimized by the pseudo-likelihood cross-validation method. To maximize the pseudo-likelihood function, an Expectation–Maximization algorithm is used. The algorithm has been tested on a dataset with more than 4900 noisy images. A comparison to other classification methods is also given.

Evaluating Stability and Comparing Output of Feature Selectors that Optimize Feature Subset Cardinality

Stability (robustness) of feature selection methods is a topic of recent interest, yet often neglected importance, with direct impact on the reliability of machine learning systems. We investigate the problem of evaluating the stability of feature selection processes yielding subsets of varying size. We introduce several novel feature selection stability measures and adjust some existing measures in a unifying framework that offers broad insight into the stability problem. We study in detail the properties of considered measures and demonstrate on various examples what information about the feature selection process can be gained. We also introduce an alternative approach to feature selection evaluation in the form of measures that enable comparing the similarity of two feature selection processes. These measures enable comparing, e.g., the output of two feature selection methods or two runs of one method with different parameters. The information obtained using the considered stability and similarity measures is shown to be usable for assessing feature selection methods (or criteria) as such.

Oscillating search algorithms for feature selection

A new sub-optimal subset search method for feature selection is introduced. As opposed to other subset selection methods the oscillating search is not dependent on pre-specified direction of search (forward or backward). The generality of the oscillating search concept allowed us to define several different algorithms suitable for different purposes. We can specify the need to obtain good results in very short time, or let the algorithm search more thoroughly to obtain near-optimum results. In many cases the oscillating search out-performed all the other tested methods. The oscillating search may be restricted by a preset time-limit, this makes it usable in real-time systems.

Conditional mutual information based feature selection for classification task

We propose a sequential forward feature selection method to find a subset of features that are most relevant to the classification task. Our approach uses novel estimation of the conditional mutual information between candidate feature and classes, given a subset of already selected features which is utilized as a classifier independent criterion for evaluation of feature subsets. The proposed mMIFS-U algorithm is applied to text classification problem and compared with MIFS method and MIFS-U method proposed by Battiti and Kwak and Choi, respectively. Our feature selection algorithm outperforms MIFS method and MIFS-U in experiments on high dimensional Reuters textual data.

Computer-Aided Evaluation of Screening Mammograms Based on Local Texture Models

We propose a new approach to diagnostic evaluation of screening mammograms based on local statistical texture models. The local evaluation tool has the form of a multivariate probability density of gray levels in a suitably chosen search window. First, the density function in the form of Gaussian mixture is estimated from data obtained by scanning of the mammogram with the search window. Then we evaluate the estimated mixture at each position and display the corresponding log-likelihood value as a gray level at the window center. The resulting log-likelihood image closely correlates with the structural details of the original mammogram and emphasizes unusual places. We assume that, in parallel use, the log-likelihood image may provide additional information to facilitate the identification of malignant lesions as untypical locations of high novelty.

Feature selection based on mutual correlation

Feature selection is a critical procedure in many pattern recognition applications. There are two distinct mechanisms for feature selection namely the wrapper methods and the filter methods. The filter methods are generally considered inferior to wrapper methods, however wrapper methods are computationally more demanding than filter methods. A novel filter feature selection method based on mutual correlation is proposed. We assess the classification performance of the proposed filter method by using the selected features to the Bayes classifier. Alternative filter feature selection methods that optimize either the Bhattacharrrya distance or the divergence are also tested. Furthermore, wrapper feature selection techniques employing several search strategies such as the sequential forward search, the oscillating search, and the sequential floating forward search are also included in the comparative study. A trade off between the classification accuracy and the feature set dimensionality is demonstrated on both two benchmark datasets from UCI repository and two emotional speech data collections.

Filter- versus wrapper-based feature selection for credit scoring

We address the problem of credit scoring as a classification and feature subset selection problem. Based on the current framework of sophisticated feature selection methods, we identify features that contain the most relevant information to distinguish good loan payers from bad loan payers. The feature selection methods are validated on several real‐world datasets with different types of classifiers. We show the advantages following from using the subspace approach to classification. We discuss many practical issues related to the applicability of feature selection methods. We show and discuss some difficulties that used to be insufficiently emphasized in standard feature selection literature.

Feature selection toolbox

A software package developed for the purpose of feature selection in statistical pattern recognition is presented. The software tool includes both several classical and new methods suitable for dimensionality reduction, classification and data representation. Examples of solved problems are given, as well as observations regarding the behavior of criterion functions.