Aritz Pérez

Sensitivity Analysis of k-Fold Cross Validation in Prediction Error Estimation

In the machine learning field, the performance of a classifier is usually measured in terms of prediction error. In most real-world problems, the error cannot be exactly calculated and it must be estimated. Therefore, it is important to choose an appropriate estimator of the error. This paper analyzes the statistical properties, bias and variance, of the k-fold cross-validation classification error estimator (k-cv). Our main contribution is a novel theoretical decomposition of the variance of the k-cv considering its sources of variance: sensitivity to changes in the training set and sensitivity to changes in the folds. The paper also compares the bias and variance of the estimator for different values of k. The experimental study has been performed in artificial domains because they allow the exact computation of the implied quantities and we can rigorously specify the conditions of experimentation. The experimentation has been performed for two classifiers (naive Bayes and nearest neighbor), different numbers of folds, sample sizes, and training sets coming from assorted probability distributions. We conclude by including some practical recommendation on the use of k-fold cross validation.

Bayesian classifiers based on kernel density estimation: Flexible classifiers

When learning Bayesian network based classifiers continuous variables are usually handled by discretization, or assumed that they follow a Gaussian distribution. This work introduces the kernel based Bayesian network paradigm for supervised classification. This paradigm is a Bayesian network which estimates the true density of the continuous variables using kernels. Besides, tree-augmented naive Bayes, k-dependence Bayesian classifier and complete graph classifier are adapted to the novel kernel based Bayesian network paradigm. Moreover, the strong consistency properties of the presented classifiers are proved and an estimator of the mutual information based on kernels is presented. The classifiers presented in this work can be seen as the natural extension of the flexible naive Bayes classifier proposed by John and Langley [G.H. John, P. Langley, Estimating continuous distributions in Bayesian classifiers, in: Proceedings of the 11th Conference on Uncertainty in Artificial Intelligence, 1995, pp. 338-345], breaking with its strong independence assumption. Flexible tree-augmented naive Bayes seems to have superior behavior for supervised classification among the flexible classifiers. Besides, flexible classifiers presented have obtained competitive errors compared with the state-of-the-art classifiers.

Supervised classification with conditional Gaussian networks: Increasing the structure complexity from naive Bayes

Most of the Bayesian network-based classifiers are usually only able to handle discrete variables. However, most real-world domains involve continuous variables. A common practice to deal with continuous variables is to discretize them, with a subsequent loss of information. This work shows how discrete classifier induction algorithms can be adapted to the conditional Gaussian network paradigm to deal with continuous variables without discretizing them. In addition, three novel classifier induction algorithms and two new propositions about mutual information are introduced. The classifier induction algorithms presented are ordered and grouped according to their structural complexity: naive Bayes, tree augmented naive Bayes, k-dependence Bayesian classifiers and semi naive Bayes. All the classifier induction algorithms are empirically evaluated using predictive accuracy, and they are compared to linear discriminant analysis, as a continuous classic statistical benchmark classifier. Besides, the accuracies for a set of state-of-the-art classifiers are included in order to justify the use of linear discriminant analysis as the benchmark algorithm. In order to understand the behavior of the conditional Gaussian network-based classifiers better, the results include bias-variance decomposition of the expected misclassification rate. The study suggests that semi naive Bayes structure based classifiers and, especially, the novel wrapper condensed semi naive Bayes backward, outperform the behavior of the rest of the presented classifiers. They also obtain quite competitive results compared to the state-of-the-art algorithms included.

A general framework for the statistical analysis of the sources of variance for classification error estimators

Estimating the prediction error of classifiers induced by supervised learning algorithms is important not only to predict its future error, but also to choose a classifier from a given set (model selection). If the goal is to estimate the prediction error of a particular classifier, the desired estimator should have low bias and low variance. However, if the goal is the model selection, in order to make fair comparisons the chosen estimator should have low variance assuming that the bias term is independent from the considered classifier. This paper follows the analysis proposed in [1] about the statistical properties of k-fold cross-validation estimators and extends it to the most popular error estimators: resubstitution, holdout, repeated holdout, simple bootstrap and 0.632 bootstrap estimators, without and with stratification. We present a general framework to analyze the decomposition of the variance of different error estimators considering the nature of the variance (irreducible/reducible variance) and the different sources of sensitivity (internal/external sensitivity). An extensive empirical study has been performed for the previously mentioned estimators with naive Bayes and C4.5 classifiers over training sets obtained from assorted probability distributions. The empirical analysis consists of decomposing the variances following the proposed framework and checking the independence assumption between the bias and the considered classifier. Based on the obtained results, we propose the most appropriate error estimations for model selection under different experimental conditions.

Using Multidimensional Bayesian Network Classifiers to Assist the Treatment of Multiple Sclerosis

Multiple sclerosis is an autoimmune disorder of the central nervous system and potentially the most common cause of neurological disability in young adults. The clinical disease course is highly variable and different multiple sclerosis subtypes can be defined depending on the progression of the severity of the disease. In the early stages, the disease subtype is unknown, and there is no information about how the severity is going to evolve. As there are different treatment options available depending on the progression of the disease, early identification has become highly relevant. Thus, given a new patient, it is important to diagnose the disease subtype. Another relevant information to predict is the expected time to reach a severity level indicating that assistance for walking is required. Given that we have to predict two correlated class variables: disease subtype and time to reach certain severity level, we use multidimensional Bayesian network classifiers because they can model and exploit the relations among both variables. Besides, the obtained models can be validated by the physicians using their expert knowledge due to the interpretability of Bayesian networks. The learning of the classifiers is made by means of a novel multiobjective approach which tries to maximize the accuracy of both class variables simultaneously. The application of the methodology proposed in this paper can help a physician to identify the expected progression of the disease and to plan the most suitable treatment.

Evaluating machine-learning techniques for recruitment forecasting of seven North East Atlantic fish species

The effect of different factors (spawning biomass, environmental conditions) on recruitment is a subject of great importance in the management of fisheries, recovery plans and scenario exploration. In this study, recently proposed supervised classification techniques, tested by the machine-learning community, are applied to forecast the recruitment of seven fish species of North East Atlantic (anchovy, sardine, mackerel, horse mackerel, hake, blue whiting and albacore), using spawning, environmental and climatic data. In addition, the use of the probabilistic flexible naive Bayes classifier (FNBC) is proposed as modelling approach in order to reduce uncertainty for fisheries management purposes. Those improvements aim is to improve probability estimations of each possible outcome (low, medium and high recruitment) based in kernel density estimation, which is crucial for informed management decision making with high uncertainty. Finally, a comparison between goodness-of-fit and generalization power is provided, in order to assess the reliability of the final forecasting models. It is found that in most cases the proposed methodology provides useful information for management whereas the case of horse mackerel is an example of the limitations of the approach. The proposed improvements allow for a better probabilistic estimation of the different scenarios, i.e. to reduce the uncertainty in the provided forecasts.

An incremental and hierarchical k-NN classifier for handwritten characters

This paper analyses the application of hierarchical classifiers based on the k-NN rule to the automatic classification of handwritten characters. The discriminating capacity of a k-NN classifier increases as the size of the reference pattern set (RPS) increases. This supposes a problem for k-NN classifiers in real applications: the high computational cost required when the RPS is large. In order to accelerate the process of calculating the distance to each pattern of the RPS, some authors propose the use of condensing techniques. These methods try to reduce the size of the RPS without losing classification power. Our alternative proposal is based on incremental learning and hierarchical classifiers with rejection techniques that reduce the computational cost of the classifier. We have used 133,944 characters (72,105 upper-case characters and 61,839 lower-case characters) of the NIST Special Data Bases 3 and 7 as experimental data set. The binary image of the character is transformed to a gray image. The best non-hierarchical classifier achieves a hit rate of 94.92% (upper-case) and 87,884% (lower-case). The hierarchical classifier achieves the same hit ratio, but with 3 times lower computational cost than the cost of the best non-hierarchical classifier found in our experimentation and 14% less than Harts (1968) algorithm.

Multidimensional multistage k-NN classifiers for handwritten digit recognition

This paper analyses the application of multistage classifiers based on the k-NN rule to the automatic classification of handwritten digits. The discriminating capacity of a k-NN classifier increases as the size and dimensionality of the reference pattern set (RPS) increases. This supposes a problem for k-NN classifiers in real applications: the high computational cost required. In order to accelerate the process of calculating the distance to each pattern of the RPS, some authors propose the use of condensing techniques. These methods try to reduce the size of the RPS without losing classification power. Our alternative proposal is based on hierarchical classifiers with rejection techniques and incremental learning that reduce the computational cost of the classifier. We have used 270,000 digits (160,000 digits for training and 110, 000 for the test) of the NIST Special Data Bases 19 and 3 (SD19 and SD3) as experimental data sets. The best non -hierarchical classifier achieves a hit rate of 99.50%. The hierarchical classifier achieves the same hit ratio, but with 24.5 times lower computational cost than best non-hierarchical classifier found in our experimentation and 6 times lower than Harts Algorithm.

Learning Maximum Weighted (k+1)-Order Decomposable Graphs by Integer Linear Programming

This work is focused on learning maximum weighted graphs subject to three structural constraints: (1) the graph is decomposable, (2) it has a maximum clique size of k + 1, and (3) it is coarser than a given maximum k-order decomposable graph. After proving that the problem is NP-hard we give a formulation of the problem based on integer linear programming. The approach has shown competitive experimental results in artificial domains. The proposed formulation has important applications in the field of probabilistic graphical models, such as learning decomposable models based on decomposable scores (e.g. log-likelihood, BDe, MDL, just to name a few).