Arnak S. Dalalyan

Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity

We study the problem of aggregation under the squared loss in the model of regression with deterministic design. We obtain sharp PAC-Bayesian risk bounds for aggregates defined via exponential weights, under general assumptions on the distribution of errors and on the functions to aggregate. We then apply these results to derive sparsity oracle inequalities.

Aggregation by exponential weighting and sharp oracle inequalities

In the present paper, we study the problem of aggregation under the squared loss in the model of regression with deterministic design. We obtain sharp oracle inequalities for convex aggregates defined via exponential weights, under general assumptions on the distribution of errors and on the functions to aggregate. We show how these results can be applied to derive a sparsity oracle inequality.

Image denoising with patch-based PCA: local versus global

In recent years, overcomplete dictionaries combined with sparse learning techniques became extremely popular in computer vision. While their usefulness is undeniable, the improvement they provide in specific tasks of computer vision is still poorly understood. The aim of the present work is to demonstrate that for the task of image denoising, nearly state-of-the-art results can be achieved using orthogonal dictionaries only, provided that they are learned directly from the noisy image. To this end, we introduce three patchbased denoising algorithms which perform hard thresholding on the coefficients of the patches in image-specific orthogonal dictionaries. The algorithms differ by the methodology of learning the dictionary: local PCA, hierarchical PCA and global PCA.We carry out a comprehensive empirical evaluation of the performance of these algorithms in terms of accuracy and running times. The results reveal that, despite its simplicity, PCA-based denoising appears to be competitive with the state-of-the-art denoising algorithms, especially for large images and moderate signal-to-noise ratios.

On the Prediction Performance of the Lasso

Although the Lasso has been extensively studied, the relationship between its prediction performance and the correlations of the covariates is not fully understood. In this paper, we give new insights into this relationship in the context of multiple linear regression. We show, in particular, that the incorporation of a simple correlation measure into the tuning parameter can lead to a nearly optimal prediction performance of the Lasso even for highly correlated covariates. However, we also reveal that for moderately correlated covariates, the prediction performance of the Lasso can be mediocre irrespective of the choice of the tuning parameter. We finally show that our results also lead to near-optimal rates for the least-squares estimator with total variation penalty.

Tight conditions for consistency of variable selection in the context of high dimensionality

We address the issue of variable selection in the regression model with very high ambient dimension, that is, when the number of variables is very large. The main focus is on the situation where the number of relevant variables, called intrinsic dimension, is much smaller than the ambient dimension d. Without assuming any parametric form of the underlying regression function, we get tight conditions making it possible to consistently estimate the set of relevant variables. These conditions relate the intrinsic dimension to the ambient dimension and to the sample size. The procedure that is provably consistent under these tight conditions is based on comparing quadratic functionals of the empirical Fourier coefficients with appropriately chosen threshold values. The asymptotic analysis reveals the presence of two quite different re gimes. The first regime is when the intrinsic dimension is fixed. In this case the situation in nonparametric regression is the same as in linear regression, that is, consistent variable selection is possible if and only if log d is small compared to the sample size n. The picture is different in the second regime, that is, when the number of relevant variables denoted by s tends to infinity as

Sharp oracle inequalities for aggregation of affine estimators

n\to\infty

PENALIZED MAXIMUM LIKELIHOOD AND SEMIPARAMETRIC SECOND-ORDER EFFICIENCY

. Then we prove that consistent variable selection in nonparametric set-up is possible only if s+loglog d is small compared to log n. We apply these results to derive minimax separation rates for the problem of variable

Mirror averaging with sparsity priors

We consider the problem of combining a (possibly uncountably infinite) set of affine estimators in non-parametric regression model with heteroscedastic Gaussian noise. Focusing on the exponentially weighted aggregate, we prove a PAC-Bayesian type inequality that leads to sharp oracle inequalities in discrete but also in continuous settings. The framework is general enough to cover the combinations of various procedures such as least square regression, kernel ridge regression, shrinking estimators and many other estimators used in the literature on statistical inverse problems. As a consequence, we show that the proposed aggregate provides an adaptive estimator in the exact minimax sense without neither discretizing the range of tuning parameters nor splitting the set of observations. We also illustrate numerically the good performance achieved by the exponentially weighted aggregate.

Sharp adaptive estimation of the drift function for ergodic diffusions

We consider the problem of estimation of a shift parameter of an unknown symmetric function in Gaussian white noise. We introduce a notion of semiparametric second-order efficiency and propose estimators that are semiparametrically efficient and second-orderefficient in our model. These estimators are of a penalized maximum likelihood type with an appropriately chosen penalty. We argue that secondorder efficiency is crucial in semiparametric problems sinceonly the second-order terms in asymptotic expansion for the risk account for the behavior of the “nonparametric component” of a semiparametric procedure, and they are not dramatically smaller than the first-order terms. 1. Introduction. Semiparametric statistical models are the ones containing a finite-dimensional parameter of interest θ and an infinite-dimensional nuisance parameter f which is a member of some large functional class. The goal is then to estimate θ efficiently without knowingf. A comprehensive account of the theory of semiparametric estimation is given in the book of Bickel, Klaassen, Ritov and Wellner [3]. In particular, it is shown that for many semiparametric models there exist estimators attaining the same asymptotic performance as efficient parametric estimators c for the problem where f is completely specified. In other words, for such semiparametric models there is no loss of efficiency as compared tothe corresponding parametric models with known f. These semiparametric models are usually called adaptive, but we prefer here to call them S-adaptive, or semiparametrically adaptive, in order to avoid confusion with nonparametric adaptivity to unknown smoothness of f. Estimators attaining parametric

Asymptotic statistical equivalence for ergodic diffusions: the multidimensional case

We consider the problem of aggregating the elements of a (possibly infinite) dictionary for building a decision procedure, that aims at minimizing a given criterion. Along with the dictionary, an independent identically distributed training sample is available, on which the performance of a given procedure can be tested. In a fairly general set-up, we establish an oracle inequality for the Mirror Averaging aggregate based on any prior distribution. This oracle inequality is applied in the context of sparse coding for different problems of statistics and machine learning such as regression, density estimation and binary classification.