Rocío Alaiz-Rodríguez

A unifying view on dataset shift in classification

The field of dataset shift has received a growing amount of interest in the last few years. The fact that most real-world applications have to cope with some form of shift makes its study highly relevant. The literature on the topic is mostly scattered, and different authors use different names to refer to the same concepts, or use the same name for different concepts. With this work, we attempt to present a unifying framework through the review and comparison of some of the most important works in the literature.

Class distribution estimation based on the Hellinger distance

Class distribution estimation (quantification) plays an important role in many practical classification problems. Firstly, it is important in order to adapt the classifier to the operational conditions when they differ from those assumed in learning. Additionally, there are some real domains where the quantification task is itself valuable due to the high variability of the class prior probabilities. Our novel quantification approach for two-class problems is based on distributional divergence measures. The mismatch between the test data distribution and validation distributions generated in a fully controlled way is measured by the Hellinger distance in order to estimate the prior probability that minimizes this divergence. Experimental results on several binary classification problems show the benefits of this approach when compared to such approaches as counting the predicted class labels and other methods based on the classifier confusion matrix or on posterior probability estimations. We also illustrate these techniques as well as their robustness against the base classifier performance (a neural network) with a boar semen quality control setting. Empirical results show that the quantification can be conducted with a mean absolute error lower than 0.008, which seems very promising in this field.

Q-Probabilistic Routing in Wireless Sensor Networks

Unpredictable topology changes, energy constraints and link unreliability make the information transmission a challenging problem in wireless sensor networks (WSN). Taking some ideas from machine learning methods, we propose a novel geographic routing algorithm for WSN, named Q-probabilistic routing (Q-PR), that makes intelligent routing decisions from the delayed reward of previous actions and the local interaction among neighbor nodes, by using reinforcement learning and a Bayesian decision model. Moreover, by considering the message importance embedded in the message itself routing decisions can be adapted to traffic importance. Experimental results show that Q-PR becomes a routing policy that, as a function of the message importance, achieves a trade-off among the expected number of retransmissions (ETX), the successful delivery rate and the network lifetime.

Local estimation of posterior class probabilities to minimize classification errors

Decision theory shows that the optimal decision is a function of the posterior class probabilities. More specifically, in binary classification, the optimal decision is based on the comparison of the posterior probabilities with some threshold. Therefore, the most accurate estimates of the posterior probabilities are required near these decision thresholds. This paper discusses the design of objective functions that provide more accurate estimates of the probability values, taking into account the characteristics of each decision problem. We propose learning algorithms based on the stochastic gradient minimization of these loss functions. We show that the performance of the classifier is improved when these algorithms behave like sample selectors: samples near the decision boundary are the most relevant during learning.

Assessing the impact of changing environments on classifier performance

The purpose of this paper is to test the hypothesis that simple classifiers are more robust to changing environments than complex ones. We propose a strategy for generating artificial, but realistic domains, which allows us to control the changing environment and test a variety of situations. Our results suggest that evaluating classifiers on such tasks is not straightforward since the changed environment can yield a simpler or more complex domain. We propose a metric capable of taking this issue into consideration and evaluate our classifiers using it. We conclude that in mild cases of population drifts simple classifiers deteriorate more than complex ones and that in more severe cases as well as in class definition changes, all classifiers deteriorate to about the same extent. This means that in all cases, complex classifiers remain more accurate than simpler ones, thus challenging the hypothesis that simple classifiers are more robust to changing environments than complex ones.

Texture and moments-based classification of the acrosome integrity of boar spermatozoa images

The automated assessment of the sperm quality is an important challenge in the veterinary field. In this paper, we explore how to describe the acrosomes of boar spermatozoa using image analysis so that they can be automatically categorized as intact or damaged. Our proposal aims at characterizing the acrosomes by means of texture features. The texture is described using first order statistics and features derived from the co-occurrence matrix of the image, both computed from the original image and from the coefficients yielded by the Discrete Wavelet Transform. Texture descriptors are evaluated and compared with moments-based descriptors in terms of the classification accuracy they provide. Experimental results with a Multilayer Perceptron and the k-Nearest Neighbours classifiers show that texture descriptors outperform moment-based descriptors, reaching an accuracy of 94.93%, which makes this approach very attractive for the veterinarian community.

Minimax classifiers based on neural networks

The problem of designing a classifier when prior probabilities are not known or are not representative of the underlying data distribution is discussed in this paper. Traditional learning approaches based on the assumption that class priors are stationary lead to sub-optimal solutions if there is a mismatch between training and future (real) priors. To protect against this uncertainty, a minimax approach may be desirable. We address the problem of designing a neural-based minimax classifier and propose two different algorithms: a learning rate scaling algorithm and a gradient-based algorithm. Experimental results show that both succeed in finding the minimax solution and it is also pointed out the differences between common approaches to cope with this uncertainty in priors and the minimax classifier.

Class and subclass probability re-estimation to adapt a classifier in the presence of concept drift

Abstract We consider the problem of classification in environments where training and test data may come from different probability distributions. When the fundamental stationary distribution assumption made in supervised learning (and often not satisfied in practice) does not hold, the classifier performance may significantly deteriorate. Several proposals have been made to deal with classification problems where the class priors change after training, but they may fail when the class conditional data densities also change. To cope with this problem, we propose an algorithm that uses unlabeled test data to adapt the classifier outputs to new operating conditions, without re-training it. The algorithm is based on a posterior probability model with two main assumptions: (1) the classes may be decomposed in several (unknown) subclasses, and (2) all changes in data distributions arise from changes in prior subclass probabilities. Experimental results with a neural network model on synthetic and remote sensing practical settings show that the adaptation at the subclass level can get a better adjustment to the new operational conditions than the methods based on class prior changes.

Cost-sensitive learning based on Bregman divergences

This paper analyzes the application of a particular class of Bregman divergences to design cost-sensitive classifiers for multiclass problems. We show that these divergence measures can be used to estimate posterior probabilities with maximal accuracy for the probability values that are close to the decision boundaries. Asymptotically, the proposed divergence measures provide classifiers minimizing the sum of decision costs in non-separable problems, and maximizing a margin in separable MAP problems.

A Visualization-Based Exploratory Technique for Classifier Comparison with Respect to Multiple Metrics and Multiple Domains

Classifier performance evaluation typically gives rise to a multitude of results that are difficult to interpret. On the one hand, a variety of different performance metrics can be applied, each adding a little bit more information about the classifiers than the others; and on the other hand, evaluation must be conducted on multiple domains to get a clear view of the classifiers general behaviour.