Michał Woźniak

Hybrid Artificial Intelligent Systems

This paper deals with discovering frequent sets for quantitative association rules mining with preserved privacy. It focuses on privacy preserving on an individual level, when true individual values, e.g., values of attributes describing customers, are not revealed. Only distorted values and parameters of the distortion procedure are public. However, a miner can discover hidden knowledge, e.g., association rules, from the distorted data. In order to find frequent sets for quantitative association rules mining with preserved privacy, not only does a miner need to discretise continuous attributes, transform them into binary attributes, but also, after both discretisation and binarisation, the calculation of the distortion parameters for new attributes is necessary. Then a miner can apply either MASK (Mining Associations with Secrecy Konstraints) or MMASK (Modified MASK) to find candidates for frequent sets and estimate their supports. In this paper the methodology for calculating distortion parameters of newly created attributes after both discretisation and binarisation of attributes for quantitative association rules mining has been proposed. The new application of MMASK for finding frequent sets in discovering quantitative association rules with preserved privacy has been also presented. The application of MMASK scheme for frequent sets mining in quantitative association rules discovery on real data sets has been experimentally verified. The results of the experiments show that both MASK and MMASK can be applied in frequent sets mining for quantitative association rules with preserved privacy, however, MMASK gives better results in this task.

Combined classifier based on feature space partitioning

This paper presents a significant modification to the AdaSS (Adaptive Splitting and Selection) algorithm, which was developed several years ago. The method is based on the simultaneous partitioning of the feature space and an assignment of a compound classifier to each of the subsets. The original version of the algorithm uses a classifier committee and a majority voting rule to arrive at a decision. The proposed modification replaces the fairly simple fusion method with a combined classifier, which makes a decision based on a weighted combination of the discriminant functions of the individual classifiers selected for the committee. The weights mentioned above are dependent not only on the classifier identifier, but also on the class number. The proposed approach is based on the results of previous works, where it was proven that such a combined classifier method could achieve significantly better results than simple voting systems. The proposed modification was evaluated through computer experiments, carried out on diverse benchmark datasets. The results are very promising in that they show that, for most of the datasets, the proposed method outperforms similar techniques based on the clustering and selection approach.

Analyzing the oversampling of different classes and types of examples in multi-class imbalanced datasets

Canonical machine learning algorithms assume that the number of objects in the considered classes are roughly similar. However, in many real-life situations the distribution of examples is skewed since the examples of some of the classes appear much more frequently. This poses a difficulty to learning algorithms, as they will be biased towards the majority classes. In recent years many solutions have been proposed to tackle imbalanced classification, yet they mainly concentrate on binary scenarios. Multi-class imbalanced problems are far more difficult as the relationships between the classes are no longer straightforward. Additionally, one should analyze not only the imbalance ratio but also the characteristics of the objects within each class. In this paper we present a study on oversampling for multi-class imbalanced datasets that focuses on the analysis of the class characteristics. We detect subsets of specific examples in each class and fix the oversampling for each of them independently. Thus, we are able to use information about the class structure and boost the more difficult and important objects. We carry an extensive experimental analysis, which is backed-up with statistical analysis, in order to check when the preprocessing of some types of examples within a class may improve the indiscriminate preprocessing of all the examples in all the classes. The results obtained show that oversampling concrete types of examples may lead to a significant improvement over standard multi-class preprocessing that do not consider the importance of example types. Graphical abstractDisplay Omitted HighlightsA thorough analysis of oversampling for handling multi-class imbalanced datasets.Proposition to detect underlying structures and example types in considered classes.Smart oversampling based on extracted knowledge about imbalance distribution types.In-depth insight into the importance of selecting proper examples for oversampling.Guidelines that allow to design efficient classifiers for multi-class imbalanced data.

On the usefulness of one-class classifier ensembles for decomposition of multi-class problems

Multi-class classification can be addressed in a plethora of ways. One of the most promising research directions is applying the divide and conquer rule, by decomposing the given problem into a set of simpler sub-problems and then reconstructing the original decision space from local responses.In this paper, we propose to investigate the usefulness of applying one-class classifiers to this task, by assigning a dedicated one-class descriptor to each class, with three main approaches: one-versus-one, one-versus-all and trained fusers. Despite not using all the knowledge available, one-class classifiers display several desirable properties that may be of benefit to the decomposition task. They can adapt to the unique properties of the target class, trying to fit a best concept description. Thus they are robust to many difficulties embedded in the nature of data, such as noise, imbalanced or complex distribution. We analyze the possibilities of applying an ensemble of one-class methods to tackle multi-class problems, with a special attention paid to the final stage - reconstruction of the original multi-class problem. Although binary decomposition is more suitable for most standard datasets, we identify the specific areas of applicability for one-class classifier decomposition.To do so, we develop a double study: first, for a given fusion method, we compare one-class and binary classifiers to find the correlations between classifier models and fusion algorithms. Then, we compare the best methods from each group (one-versus-one, one-versus-all and trained fusers) to draw conclusions about the overall performance of one-class solutions. We show, backed-up by thorough statistical analysis, that one-class decomposition is a worthwhile approach, especially in case of problems with complex distribution and a large number of classes. Graphical abstractDisplay Omitted HighlightsA novel approach for tackling multi-class problems with ensemble of one-class classifiers.A thorough analysis on areas of applicability of one-class decomposition.Identification of problems, for which one-class decomposition is superior to binarization.Examination of correlations between the used fusion technique and decomposition scheme.An exhaustive experimental study that gives a detailed outlook on the quality of investigated methods.

One-class classifiers with incremental learning and forgetting for data streams with concept drift

One of the most important challenges for machine learning community is to develop efficient classifiers which are able to cope with data streams, especially with the presence of the so-called concept drift. This phenomenon is responsible for the change of classification task characteristics, and poses a challenge for the learning model to adapt itself to the current state of the environment. So there is a strong belief that one-class classification is a promising research direction for data stream analysis—it can be used for binary classification without an access to counterexamples, decomposing a multi-class data stream, outlier detection or novel class recognition. This paper reports a novel modification of weighted one-class support vector machine, adapted to the non-stationary streaming data analysis. Our proposition can deal with the gradual concept drift, as the introduced one-class classifier model can adapt its decision boundary to new, incoming data and additionally employs a forgetting mechanism which boosts the ability of the classifier to follow the model changes. In this work, we propose several different strategies for incremental learning and forgetting, and additionally we evaluate them on the basis of several real data streams. Obtained results confirmed the usability of proposed classifier to the problem of data stream classification with the presence of concept drift. Additionally, implemented forgetting mechanism assures the limited memory consumption, because only quite new and valuable examples should be memorized.

Combining diverse one-class classifiers

Multiple Classifier Systems (MCSs) are the focus of intense research and a large variety of methods have been developed in order to exploit strengths of individual classifiers. In this paper we address the problem how to implement a multi-class classifier by an ensemble of one-class classifiers. To improve the performance of a compound classifier, different individual classifiers (which may e.g., differ in complexity, type, training algorithm or other) can be combined and that could increase its both performance, and robustness. The model of one-class classifiers is dedicated to recognize one class only, therefore it is a quite difficult to produce MCSs on the basis of it. One of the important problem is how to ensure diversity of classifier ensemble which consists of one-class classifiers. Well-known diversity measures have been developed for committees of multiclass classifiers. In this work we propose a novel diversity measure which can be applied to a set of one-class classifiers. Additionally we propose a classifier fusion model dedicated to one-class classifiers, which allows more than one classifier per class. We will try answer the question if increasing number of individual one-class classifier has an impact on quality of MCS. The proposed model was evaluated by computer experiments and their results prove that proposed model can outperform well known fusion methods.

Complexity and multithreaded implementation analysis of one class-classifiers fuzzy combiner

More recently, neural network techniques and fuzzy logic inference systems have been receiving an increasing attention. At the same time, methods of establishing decision by a group of classifiers are regarded as a general problem in various application areas of pattern recognition. Similarly to standard two-class pattern recognition methods, one-class classifiers hardly ever fit the data distribution perfectly. The paper presents fuzzy models for one-class classifier combination, compares their computational and expected space complexity with the one from ECOC and decision templates, presents possibility to speed up a fuser processing by means of multithreading.

Designing Cost-Sensitive Ensemble – Genetic Approach

The paper focuses on the problem of choosing classifiers for a committee of multiple classifier systems. We propose to design such an ensemble on the basis of an executing cost of elementary classifiers and additionally we fix mentioned above cost limit. Properties of the proposed approach were evaluated on the basis of computer experiments which were carried out on varied benchmark datasets. The results of experiments confirm that our proposition can be useful tool for designing cost-sensitive classifier committees.

Dynamic classifier selection for one-class classification

Introduction of dynamic classifier selection for one-class classification.Three novel competence measures for one-class classifiers.Gaussian approach for extending competence over the entire decision space.Results indicating that dynamic selection is a good alternative to static ensembles. One-class classification is among the most difficult areas of the contemporary machine learning. The main problem lies in selecting the model for the data, as we do not have any access to counterexamples, and cannot use standard methods for estimating the classifier quality. Therefore ensemble methods that can use more than one model, are a highly attractive solution. With an ensemble approach, we prevent the situation of choosing the weakest model and usually improve the robustness of our recognition system. However, one cannot assume that all classifiers available in the pool are in general accurate - they may have local competence areas in which they should be employed. In this work, we present a dynamic classifier selection method for constructing efficient one-class ensembles. We propose to calculate the competencies of all classifiers for a given validation example and use them to estimate their competencies over the entire decision space with the Gaussian potential function. We introduce three measures of classifiers competence designed specifically for one-class problems. Comprehensive experimental analysis, carried on a number of benchmark data and backed-up with a thorough statistical analysis prove the usefulness of the proposed approach.

Application of Combined Classifiers to Data Stream Classification

The progress of computer science caused that many institutions collected huge amount of data, which analysis is impossible by human beings. Nowadays simple methods of data analysis are not sufficient for efficient management of an average enterprize, since for smart decisions the knowledge hidden in data is highly required, as which multiple classifier systems are recently the focus of intense research. Unfortunately the great disadvantage of traditional classification methods is that they ”assume” that statistical properties of the discovered concept (which model is predicted) are being unchanged. In real situation we could observe so-called concept drift, which could be caused by changes in the probabilities of classes or/and conditional probability distributions of classes. The potential for considering new training data is an important feature of machine learning methods used in security applications or marketing departments. Unfortunately, the occurrence of this phenomena dramatically decreases classification accuracy.