Núria Macià

Preliminary approach on synthetic data sets generation based on class separability measure

Usually, performance of classifiers is evaluated on real-world problems that mainly belong to public repositories. However, we ignore the inherent properties of these data and how they affect classifier behavior. Also, the high cost or the difficulty of experiments hinder the data collection, leading to complex data sets characterized by few instances, missing values, and imprecise data. The generation of synthetic data sets solves both issues and allows us to build problems with a minor cost and whose characteristics are predefined. This is useful to test system limitations in a controlled framework. This paper proposes to generate synthetic data sets based on data complexity. We rely on the length of the class boundary to build the data sets, obtaining a preliminary set of benchmarks to assess classifier accuracy. The study can be further matured to identify regions of competence for classifiers.

Genetic-Based Synthetic Data Sets for the Analysis of Classifiers Behavior

In this paper, we highlight the use of synthetic data sets to analyze learners behavior under bounded complexity. We propose a method to generate synthetic data sets with a specific complexity, based on the length of the class boundary. We design a genetic algorithm as a search technique and find it useful to obtain class labels according to the desired complexity. The results show the suitability of the genetic algorithm as a framework to provide artificial benchmark problems that can be further enriched with the use of multi-objective and niching strategies.

A Methodology for Analyzing Case Retrieval from a Clustered Case Memory

Case retrieval from a clustered case memory consists in finding out the clusters most similar to the new input case, and then retrieving the cases from them. Although the computational time is improved, the accuracy rate may be degraded if the clusters are not representative enough due to data geometry. This paper proposes a methodology for allowing the expert to analyze the case retrieval strategies from a clustered case memory according to the required computational time improvement and the maximum accuracy reduction accepted. The mechanisms used to assess the data geometry are the complexity measures. This methodology is successfully tested on a case memory organized by a Self-Organization Map.

In search of targeted-complexity problems

Currently available real-world problems do not cover the whole complexity space and, therefore, do not allow us to thoroughly test learner behavior on the border of its domain of competence. Thus, the necessity of developing a more suitable testing scenario arises. With this in mind, data complexity analysis has shown promise in characterizing difficulty of classification problems through a set of complexity descriptors which used in artificial data sets generation could supply the required framework to refine and design learners. This paper, then, proposes the use of instance selection based on an evolutionary multiobjective technique to generate data sets that meet specific characteristics established by such complexity descriptors. These artificial targeted-complexity problems, which capture the essence of real-world structures, may help to define a set of benchmarks that contributes to test the properties of learners and to improve them.

Beyond Homemade Artificial Data Sets

One of the most important challenges in supervised learning is how to evaluate the quality of the models evolved by different machine learning techniques. Up to now, we have relied on measures obtained by running the methods on a wide test bed composed of real-world problems. Nevertheless, the unknown inherent characteristics of these problems and the bias of learners may lead to inconclusive results. This paper discusses the need to work under a controlled scenario and bets on artificial data set generation. A list of ingredients and some ideas about how to guide such generation are provided, and promising results of an evolutionary multi-objective approach which incorporates the use of data complexity estimates are presented.

EMO shines a light on the holes of complexity space

Typical domains used in machine learning analyses only cover the complexity space partially, remaining a large proportion of problem difficulties that are not tested. Since the acquisition of new real-world problems is costly, the machine learning community has started giving importance to the automatic generation of learning domains with bounded difficulty. This paper proposes the use of an evolutionary multi-objective technique to generate artificial data sets that meet specific characteristics and fill these holes. The results show that the multi-objective evolutionary algorithm is able to create data sets of different complexities, covering most of the solution space where we had no real-world problem representatives. The proposed method is the starting point to study data complexity estimates and steps forward in the gap between data and learners.