Krzysztof Grabczewski

Meta-Learning in Computational Intelligence

Computational Intelligence (CI) community has developed hundreds of algorithms for intelligent data analysis, but still many hard problems in computer vision, signal processing or text and multimedia understanding, problems that require deep learning techniques, are open. Modern data mining packages contain numerous modules for data acquisition, pre-processing, feature selection and construction, instance selection, classification, association and approximation methods, optimization techniques, pattern discovery, clusterization, visualization and post-processing. A large data mining package allows for billions of ways in which these modules can be combined. No human expert can claim to explore and understand all possibilities in the knowledge discovery process. This is where algorithms that learn how to learnl come to rescue. Operating in the space of all available data transformations and optimization techniques these algorithms use meta-knowledge about learning processes automatically extracted from experience of solving diverse problems. Inferences about transformations useful in different contexts help to construct learning algorithms that can uncover various aspects of knowledge hidden in the data. Meta-learning shifts the focus of the whole CI field from individual learning algorithms to the higher level of learning how to learn. This book defines and reveals new theoretical and practical trends in meta-learning, inspiring the readers to further research in this exciting field.

Feature selection with decision tree criterion

Classification techniques applicable to real life data are more and more often complex hybrid systems comprising feature selection. To augment their efficiency we propose two feature selection algorithms, which take advantage of a decision tree criterion. Large computational experiments have been done to test the possibilities of the two methods and to compare their results with other techniques of similar goal.

Hybrid Neural-global Minimization Method of Logical Rule Extraction

Methodology of extraction of optimal sets of logical rules using neural networks and global minimization procedures has been developed. Initial rules are extracted using density estimation neural networks with rectangular functions or multi-layered perceptron (MLP) networks trained with constrained backpropagation algorithm, transforming MLPs into simpler networks performing logical functions. A constructive algorithm called C-MLP2LN is proposed, in which rules of increasing specificity are generated consecutively by adding more nodes to the network. Neural rule extraction is followed by optimization of rules using global minimization techniques. Estimation of confidence of various sets of rules is discussed. The hybrid approach to rule extraction has been applied to a number of benchmark and real life problems with very good results.

Heterogeneous Forests of Decision Trees

In many cases it is better to extract a set of decision trees and a set of possible logical data descriptions instead of a single model. The trees that include premises with constraints on the distances from some reference points are more flexible because they provide nonlinear decision borders. Methods for creating heterogeneous forests of decision trees based on Separability of Split Value (SSV) criterion are presented. The results confirm their usefulness in understanding data structures.

Versatile and Efficient Meta-Learning Architecture: Knowledge Representation and Management in Computational Intelligence

There are many data mining systems derived from machine learning, neural network, statistics and other fields. Most of them are dedicated to some particular algorithms or applications. Unfortunately, their architectures are still too naive to provide satisfactory background for advanced meta-learning problems. In order to efficiently perform sophisticated meta-level analysis, we need a very versatile, easily expandable system (in many independent aspects), which uniformly deals with different kinds of models and models with very complex structures of models (not only committees but also much more hierarchic models). Meta-level techniques must provide mechanisms facilitating optimization of computation time and memory consumption. This article presents requirements and their motivations for an advanced data mining system, efficient not only in model construction for given data, but also in meta-learning. Some particular solutions to significant problems are presented. The newly proposed advanced meta-learning architecture has been implemented in our new data analysis system.

Heterogeneous adaptive systems

Most adaptive systems are homogenous, i.e., they are built from processing elements of the same type. MLP neural networks and decision trees use nodes that partition the input space by hyperplanes. Other types of neural networks use nodes that provide spherical or ellipsoidal decision borders. This may not be the best inductive bias for a given data, frequently requiring a large number of processing elements even in cases when simple solutions exist. In heterogeneous adaptive systems different types of decision borders are used at each stage, enabling the discovery of the most appropriate bias for the data. The neural decision tree and similarity-based systems of this kind are described here. Results from a novel heterogeneous decision tree algorithm are presented as an example of this approach.

Optimization and Interpretation of Rule-based Classifiers

Machine learning methods are frequently used to create rule-based classifiers. For continuous features linguistic variables used in conditions of the rules are defined by membership functions. These linguistic variables should be optimized at the level of single rules or sets of rules. Assuming the Gaussian uncertainty of input values allows to increase the accuracy of predictions and to estimate probabilities of different classes. Detailed interpretation of relevant rules is possible using (probabilistic) confidence intervals. A real life example of such interpretation is given for personality disorders. The approach to optimization and interpretation described here is applicable to any rule-based system.

Mining for Complex Models Comprising Feature Selection and Classification

Different classification tasks require different learning schemes to be satisfactorily solved. Most real-world datasets can be modeled only by complex structures resulting from deep data exploration with a number of different classification and data transformation methods. The search through the space of complex structures must be augmented with reliable validation strategies. All these techniques were necessary to build accurate models for the five high-dimensional datasets of the NIPS 2003 Feature Selection Challenge. Several feature selection algorithms (e.g. based on variance, correlation coefficient, decision trees) and several classification schemes (e.g. nearest neighbors, Normalized RBF, Support Vector Machines) were used to build complex models which transform the data and then classify. Committees of feature selection models and ensemble classifiers were also very helpful to construct models of high generalization abilities.

Feature selection based on information theory, consistency and separability indices

Two new feature selection methods are introduced, the first based on separability criterion, the second on a consistency index that includes interactions between the selected subsets of features. Comparison of accuracy was made against information-theory based selection methods on several datasets training neurofuzzy and nearest neighbor methods on various subsets of selected features. Methods based on separability seem to be most promising.

SSV Criterion Based Discretization for Naive Bayes Classifiers

Decision tree algorithms deal with continuous variables by finding split points which provide best separation of objects belonging to different classes. Such criteria can also be used to augment methods which require or prefer symbolic data. A tool for continuous data discretization based on the SSV criterion (designed for decision trees) has been constructed. It significantly improves the performance of Naive Bayes Classifier. The combination of the two methods has been tested on 15 datasets from UCI repository and compared with similar approaches. The comparison confirms the robustness of the system.