Gonzalo Ramos-Jiménez

Online and Non-Parametric Drift Detection Methods Based on Hoeffding’s Bounds

Incremental and online learning algorithms are more relevant in the data mining context because of the increasing necessity to process data streams. In this context, the target function may change overtime, an inherent problem of online learning (known as concept drift). In order to handle concept drift regardless of the learning model, we propose new methods to monitor the performance metrics measured during the learning process, to trigger drift signals when a significant variation has been detected. To monitor this performance, we apply some probability inequalities that assume only independent, univariate and bounded random variables to obtain theoretical guarantees for the detection of such distributional changes. Some common restrictions for the online change detection as well as relevant types of change (abrupt and gradual) are considered. Two main approaches are proposed, the first one involves moving averages and is more suitable to detect abrupt changes. The second one follows a widespread intuitive idea to deal with gradual changes using weighted moving averages. The simplicity of the proposed methods, together with the computational efficiency make them very advantageous. We use a Naive Bayes classifier and a Perceptron to evaluate the performance of the methods over synthetic and real data.

Incremental algorithm driven by error margins

Incremental learning is a good approach for classification when data-sets are too large or when new examples can arrive at any time. Forgetting these examples while keeping only the relevant information lets us reduce memory requirements. The algorithm presented in this paper, called IADEM, has been developed using these approaches and other concepts such as Chernoff and Hoeffding bounds. The most relevant features of this new algorithm are: its capability to deal with datasets of any size for inducing accurate trees and its capacity to keep updated the estimation error of the tree that is being induced. This estimation of the error is fundamental to satisfy the user requirements about the desired error in the tree and to detect noise in the datasets.

Fast Adapting Ensemble: A New Algorithm for Mining Data Streams with Concept Drift

The treatment of large data streams in the presence of concept drifts is one of the main challenges in the field of data mining, particularly when the algorithms have to deal with concepts that disappear and then reappear. This paper presents a new algorithm, called Fast Adapting Ensemble (FAE), which adapts very quickly to both abrupt and gradual concept drifts, and has been specifically designed to deal with recurring concepts. FAE processes the learning examples in blocks of the same size, but it does not have to wait for the batch to be complete in order to adapt its base classification mechanism. FAE incorporates a drift detector to improve the handling of abrupt concept drifts and stores a set of inactive classifiers that represent old concepts, which are activated very quickly when these concepts reappear. We compare our new algorithm with various well-known learning algorithms, taking into account, common benchmark datasets. The experiments show promising results from the proposed algorithm (regarding accuracy and runtime), handling different types of concept drifts.

FE-CIDIM: fast ensemble of CIDIM classifiers

An active research area in machine learning is the construction of multiple classifier systems to increase learning accuracy of simple classifiers. In this article, we present E-CIDIM, a multiple classifier system designed to improve the performance of CIDIM, and FE-CIDIM, an algorithm developed to speed up E-CIDIM. CIDIM is an algorithm that induces small and accurate decision trees. E-CIDIM keeps a maximum number of trees and induces new trees that may substitute the old trees in the ensemble. The substitution process finishes when none of the new trees improves the accuracy of any of the trees in the ensemble after a preconfigured number of attempts. FE-CIDIM has been developed to speed up the convergence of E-CIDIM using a more restrictive substitution method. We will show that the accuracy obtained thanks to a unique instance of CIDIM can be improved utilizing these new multiple classifier systems.

Online adaptive decision trees based on concentration inequalities

Classification trees are a powerful tool for mining non-stationary data streams. In these situations, massive data are constantly generated at high speed and the underlying target function can change over time. The iadem family of algorithms is based on Hoeffdings and Chernoffs bounds and induces online decision trees from data streams, but is not able to handle concept drift. This study extends this family to deal with time-changing data streams. The new online algorithm, named iadem-3, performs two main actions in response to a concept drift. Firstly, it resets the variables affected by the change and maintains unbroken the structure of the tree, which allows for changes in which consecutive target functions are very similar. Secondly, it creates alternative models that replace parts of the main tree when they significantly improve the accuracy of the model, thereby rebuilding the main tree if needed. An online change detector and a non-parametric statistical test based on Hoeffdings bounds are used to guarantee this significance. A new pruning method is also incorporated in iadem-3, making sure that all split tests previously installed in decision nodes are useful. The learning model is also viewed as an ensemble of classifiers, and predictions of the main and alternative models are combined to classify unlabeled examples. iadem-3 is empirically compared with various well-known decision tree induction algorithms for concept drift detection. We empirically show that our new algorithm often reaches higher levels of accuracy with smaller decision tree models, maintaining the processing time bounded, irrespective of the number of instances processed.

Induction of decision trees using an internal control of induction

In this paper we present CIDIM (Control of Induction by sample DIvision Method), an algorithm that has been developed to induce small and accurate decision trees using a set of examples. It uses an internal control of induction to stop the induction and to avoid the overfitting. Other ideas like a dichotomic division or groups of consecutive values are used to improve the performance of the algorithm. CIDIM has been successfully compared with ID3 and C4.5. It induces trees that are significantly better than those induced by ID3 or C4.5 in almost every experiment.

A Biomimetical Dynamic Window Approach to Navigation for Collaborative Control

Shared control is a strategy used in assistive platforms to combine human and robot orders to achieve a goal. Collaborative control is a specific shared control approach, in which users and robots commands are merged into an emergent one in a continuous way. Robot commands tend to improve efficiency and safety. However, sometimes, assistance can be rejected by users when their commands are too altered. This provokes frustration and stress and, usually, decreases emergent efficiency. To improve acceptance, robot navigation algorithms can be adapted to mimic human behavior when possible. We propose a novel variation of the well-known dynamic window approach (DWA) that we call biomimetical DWA (BDWA). The BDWA relies on a reward function extracted from real traces from volunteers presenting different motor disabilities navigating in a hospital environment using a rollator for support. We have compared the BDWA with other reactive algorithms in terms of similarity to paths completed by people with disabilities using a robotic rollator in a rehabilitation hospital unit. The BDWA outperforms all tested algorithms in terms of likeness to human paths and success rate.

ML-CIDIM: multiple layers of multiple classifier systems based on CIDIM

An active research area in Machine Learning is the construction of multiple classifier systems to increase learning accuracy of simple classifiers. In this paper we present a method to improve even more the accuracy: ML-CIDIM. This method has been developed by using a multiple classifier system which basic classifier is CIDIM, an algorithm that induces small and accurate decision trees. CIDIM makes a random division of the training set into two subsets and uses them to build an internal bound condition. ML-CIDIM induces some multiple classifier systems based on CIDIM and places them in different layers, trying to improve the accuracy of the previous layer with the following one. In this way, the accuracy obtained thanks to a unique multiple classifier system based on CIDIM can be improved. In reference to the accuracy of the classifier system built with ML-CIDIM, we can say that it competes well against bagging and boosting at statistically significant confidence levels.

Studying the hybridization of artificial neural networks in HECIC

One of the most relevant tasks concerning Machine Learning is the induction of classifiers, which can be used to classify or to predict. Those classifiers can be used in an isolated way, or can be combined to build a multiple classifier system. Building many-layered systems or knowing relation between different base classifiers are of special interest. Thus, in this paper we will use the HECIC system which consists of two layers: the first layer is a multiple classifier system that processes all the examples and tries to classify them; the second layer is an individual classifier that learns using the examples that are not unanimously classified by the first layer (incorporating new information). While using this system in a previous work we detected that some combinations that hybridize artificial neural networks (ANN) in one of the two layers seemed to get high-accuracy results. Thus, in this paper we have focused on the study of the improvement achieved by using different kinds of ANN in this two-layered system.

Hybridizing Ensemble Classifiers with Individual Classifiers

Two extensive research areas in Machine Learning are classification and prediction. Many approaches have been focused in the induction of ensemble to increase learning accuracy of individual classifiers. Recently, new approaches, different to those that look for accurate and diverse base classifiers, are emerging. In this paper we present a system made up of two layers: in the first layer, one ensemble classifier process every example and tries to classify them; in the second layer, one individual classifier is induced using the examples that are not unanimously classified by the ensemble. In addition, the examples that reach to the second layer incorporate new information added in the ensemble. Thus, we can achieve some improvement in the accuracy level, because the second layer can do more informed classifications. In the experimental section we present some results that suggest that our proposal can actually improve the accuracy of the system.