Gilles Blanchard

BCI competition 2003-data set IIa: spatial patterns of self-controlled brain rhythm modulations

A brain-computer interface (BCI) is a system that should in its ultimate form translate a subjects intent into a technical control signal without resorting to the classical neuromuscular communication channels. By using that signal to, e.g., control a wheelchair or a neuroprosthesis, a BCI could become a valuable tool for paralyzed patients. One approach to implement a BCI is to let users learn to self-control the amplitude of some of their brain rhythms as extracted from multichannel electroencephalogram. We present a method that estimates subject-specific spatial filters which allow for a robust extraction of the rhythm modulations. The effectiveness of the method was proved by achieving the minimum prediction error on data set IIa in the BCI Competition 2003, which consisted of data from three subjects recorded in ten sessions.

Statistical properties of kernel principal component analysis

The main goal of this paper is to prove inequalities on the reconstruction error for kernel principal component analysis. With respect to previous work on this topic, our contribution is twofold: (1) we give bounds that explicitly take into account the empirical centering step in this algorithm, and (2) we show that a “localized” approach allows to obtain more accurate bounds. In particular, we show faster rates of convergence towards the minimum reconstruction error; more precisely, we prove that the convergence rate can typically be faster than n−1/2. We also obtain a new relative bound on the error.A secondary goal, for which we present similar contributions, is to obtain convergence bounds for the partial sums of the biggest or smallest eigenvalues of the kernel Gram matrix towards eigenvalues of the corresponding kernel operator. These quantities are naturally linked to the KPCA procedure; furthermore these results can have applications to the study of various other kernel algorithms.The results are presented in a functional analytic framework, which is suited to deal rigorously with reproducing kernel Hilbert spaces of infinite dimension.

Hierarchical testing designs for pattern recognition

We explore the theoretical foundations of a twenty questions approach to pattern recognition. The object of the analysis is the computational process itself rather than probability distributions (Bayesian inference) or decision boundaries (statistical learning). Our formulation is motivated by applications to scene interpretation in which there are a great many possible explanations for the data, one (background) is statistically dominant, and it is imperative to restrict intensive computation to genuinely ambiguous regions. The focus here is then on pattern filtering: Given a large set Y of possible patterns or explanations, narrow down the true one Y to a small (random) subset Y ⊂ Y, of detected patterns to be subjected to further, more intense, processing. To this end, we consider a family of hypothesis tests for Y e A versus the nonspecific alternatives Y ∈ A c . Each test has null type I error and the candidate sets A ⊂ Y are arranged in a hierarchy of nested partitions. These tests are then characterized by scope (|A|), power (or type II error) and algorithmic cost. We consider sequential testing strategies in which decisions are made iteratively, based on past outcomes, about which test to perform next and when to stop testing. The set Y is then taken to be the set of patterns that have not been ruled out by the tests performed. The total cost of a strategy is the sum of the testing cost and the postprocessing cost (proportional to |Y|) and the corresponding optimization problem is analyzed. As might be expected, under mild assumptions good designs for sequential testing strategies exhibit a steady progression from broad scope coupled with low power to high power coupled with dedication to specific explanations. In the assumptions ensuing this property a key role is played by the ratio cost/power. These ideas are illustrated in the context of detecting rectangles amidst clutter.

Optimal dyadic decision trees

We introduce a new algorithm building an optimal dyadic decision tree (ODT). The method combines guaranteed performance in the learning theoretical sense and optimal search from the algorithmic point of view. Furthermore it inherits the explanatory power of tree approaches, while improving performance over classical approaches such as CART/C4.5, as shown on experiments on artificial and benchmark data.

Exemplar-based image inpainting: Fast priority and coherent nearest neighbor search

Greedy exemplar-based algorithms for inpainting face two main problems, decision of filling-in order and selection of good exemplars from which the missing region is synthesized. We propose an algorithm that tackle these problems with improvements in the preservation of linear edges, and reduction of error propagation compared to well-known algorithms from the literature. Our improvement in the filling-in order is based on a combination of priority terms, previously defined by Criminisi, that better encourages the early synthesis of linear structures. The second contribution helps reducing the error propagation thanks to a better detection of outliers from the candidate patches carried. This is obtained with a new metric that incorporates the whole information of the candidate patches. Moreover, our proposal has significant lower computational load than most of the algorithms used for comparison in this paper.

Discrepancy principle for statistical inverse problems with application to conjugate gradient iteration

The authors discuss the use of the discrepancy principle for statistical inverse problems, when the underlying operator is of trace class. Under this assumption the discrepancy principle is well defined, however a plain use of it may occasionally fail and it will yield sub-optimal rates. Therefore, a modification of the discrepancy is introduced, which corrects both of the above deficiencies. For a variety of linear regularization schemes as well as for conjugate gradient iteration it is shown to yield order optimal a priori error bounds under general smoothness assumptions. A posteriori error control is also possible, however at a sub-optimal rate, in general. This study uses and complements previous results for bounded deterministic noise.

Some nonasymptotic results on resampling in high dimension, I: Confidence regions

We study generalized bootstrap confidence regions for the mean of a random vector whose coordinates have an unknown dependency structure. The random vector is supposed to be either Gaussian or to have a symmetric and bounded distribution. The dimensionality of the vector can possibly be much larger than the number of observations and we focus on a non-asymptotic control of the confidence level, following ideas inspired by recent results in learning theory. We consider two approaches, the first based on a concentration principle (valid for a large class of resampling weights) and the second on a direct resampled quantile, specifically using Rademacher weights. Several intermediate results established in the approach based on concentration principles are of self-interest. We also discuss the question of accuracy when using Monte-Carlo approximations of the resampled quantities. We present an application of these results to the one-sided and two-sided multiple testing problem, in which we derive several resampling-based step-down procedures providing a non-asymptotic FWER control. We compare our different procedures in a simulation study, and we show that they can outperform Bonferronis or Holms procedures as soon as the observed vector has sufficiently correlated coordinates.

Oracle Bounds and Exact Algorithm for Dyadic Classification Trees

This paper introduces a new method using dyadic decision trees for estimating a classification or a regression function in a multiclass classification problem. The estimator is based on model selection by penalized empirical loss minimization. Our work consists in two complementary parts: first, a theoretical analysis of the method leads to deriving oracle-type inequalities for three different possible loss functions. Secondly, we present an algorithm able to compute the estimator in an exact way.

Discussion: Local Rademacher complexities and oracle inequalities in risk minimization

In this magnificent paper, Professor Koltchinskii offers general and powerful performance bounds for empirical risk minimization, a fundamental principle of statistical learning theory. Since the elegant pioneering work of Vapnik and Chervonenkis in the early 1970s, various such bounds have been known that relate the performance of empirical risk minimizers to combinatorial and geometrical features of the class over which the minimization is performed. This area of research has been a rich source of motivation and a major field of applications of empirical process theory. The appearance of advanced concentration inequalities in the 1990s, primarily thanks to Talagrand’s influential work, provoked major advances in both empirical process theory and statistical learning theory and led to a much deeper understanding of some of the basic phenomena. In the discussed paper Professor Koltchinskii develops a powerful new methodology, iterative localization, which, with the help of concentration inequalities, is able to explain most of the recent results and go significantly beyond them in many cases. The main motivation behind Professor Koltchinskii’s paper is based on classical problems of statistical learning theory such as binary classification and regression in which, given a sample (Xi ,Y i), i = 1 ,...,n , of independent and identically distributed pairs of random variables (where the Xi take their values in some feature space X and the Yi are, say, real-valued), the goal is to find a function f : X → R whose risk, defined in terms of the expected value of an appropriately chosen loss function, is as small as possible. In the remaining part of this discussion we point out how the performance bounds of Professor Koltchinskii’s paper can be used to study a seemingly different model, motivated by nonparametric ranking problems, which has received increasing attention both in the statistical and machine learning literature. Indeed, in several applications, such as the search engine problem or credit risk screening, the goal is to learn how to rank—or to score—observations rather than just classify them. In this case, performance measures involve pairs of observations, as can be seen, for instance, with the AUC (Area Under an ROC Curve) criterion. In thisThese last years, much attention has been paid to the construction of model selection criteria via penalization. Vladimir Koltchinskii has to be congratulated for providing a theory reaching a level of generality that is sufficiently high to recover most of the recent results obtained on this topic in the context of statistical learning. Thanks to concentration inequalities and empirical process theory, we are now at a point where the problem of understanding what is the order of the excess risk for the empirical minimizer on a given model is elucidated. Koltchinskii’s paper provides several ways of expressing that this excess risk can be sharply bounded by quantities depending on the complexity of the model in various senses. The most prominent relies on Rademacher processes, which Vladimir Koltchinskii himself pioneered in introducing in statistics. We even know that these upper bounds on the excess risk are often unimprovable (see the lower bounds in [6], e.g.). The same machinery used to analyze the excess risk can be applied to produce penalized criteria and to establish oracle-type risk bounds for the sodefined penalized empirical risk minimizer. The problem of defining properly penalized criteria is particularly challenging in the classification context, since it is connected to the question of defining optimal classifiers without knowing in advance the “noise condition” of the underlying distribution [(8.2) of the discussed paper]. This condition determines the attainable rates of convergence and is a topic attracting much attention in the statistical learning community at this moment (see the numerous references in the discussed paper). What we would like to discuss is the gap between theory and practice of model selection. Of course, the existence of a gap between the methods which are analyzed in theory, and those which are used in practice, is in

Finite-Dimensional Projection for Classification and Statistical Learning

In this paper, a new method for the binary classification problem is studied. It relies on empirical minimization of the hinge risk over an increasing sequence of finite-dimensional spaces. A suitable dimension is picked by minimizing the regularized risk, where the regularization term is proportional to the dimension. An oracle-type inequality is established for the excess generalization risk (i.e., regret to Bayes) of the procedure, which ensures adequate convergence properties of the method. We suggest to select the considered sequence of subspaces by applying kernel principal components analysis (KPCA). In this case, the asymptotical convergence rate of the method can be better than what is known for the support vector machine (SVM). Exemplary experiments are presented on benchmark data sets where the practical results of the method are comparable to the SVM.