László Györfi

Distribution estimation consistent in total variation and in two types of information divergence

The problem of the nonparametric estimation of a probability distribution is considered from three viewpoints: the consistency in total variation, the consistency in information divergence, and consistency in reversed-order information divergence. These types of consistencies are relatively strong criteria of convergence, and a probability distribution cannot be consistently estimated in either type of convergence without any restrictions on the class of probability distributions allowed. Histogram-based estimators of distribution are presented which, under certain conditions, converge in total variation, in information divergence, and in reversed-order information divergence to the unknown probability distribution. Some a priori information about the true probability distribution is assumed in each case. As the concept of consistency in information divergence is stronger than that of convergence in total variation, additional assumptions are imposed in the cases of informational divergences. >

Nonparametric Kernel-Based Sequential Investment Strategies

The purpose of this paper is to introduce sequential investment strategies that guarantee an optimal rate of growth of the capital, under minimal assumptions on the behavior of the market. The new strategies are analyzed both theoretically and empirically. The theoretical results show that the asymptotic rate of growth matches the optimal one that one could achieve with a full knowledge of the statistical properties of the underlying process generating the market, under the only assumption that the market is stationary and ergodic. The empirical results show that the performance of the proposed investment strategies measured on past nyse and currency exchange data is solid, and sometimes even spectacular.

Density-free convergence properties of various estimators of entropy

Abstract Let ƒ(x) be a probability density function, x∈Rd. The Shannon (or differential) entropy is defined as H(ƒ)=−∫ƒ(x) log ƒ(x) d x . In this paper we propose, based on a random sample X1,…, Xn generated from ƒ, two new nonparametric estimators for H(ƒ) . Both entropy estimators are histogram-based in the sense that they involve a histogram-based desntiy estimator ƒ n . We prove their a.s. consistency with the only condition on ƒ that H(ƒ) is finite.

There is no universal source code for an infinite source alphabet

Shows that a discrete infinite distribution with finite entropy cannot be estimated consistently in information divergence. As a corollary the authors show that there is no universal source code for an infinite source alphabet over the class of all discrete memoryless sources with finite entropy. >

Kernel-based semi-log-optimal empirical portfolio selection strategies

The purpose of this paper is to introduce an approximation of the kernel-based log-optimal investment strategy that guarantees an almost optimal rate of growth of the capital under minimal assumptions on the behavior of the market. The new strategy uses much less knowledge on the distribution of the market process. It is analyzed both theoretically and empirically. The theoretical results show that the asymptotic rate of growth well approximates the optimal one that one could achieve with a full knowledge of the statistical properties of the underlying process generating the market, under the only assumption that the market is stationary and ergodic. The empirical results show that the proposed semi-log-optimal and the log-optimal strategies have essentially the same performance measured on past NYSE data.

Principles of nonparametric learning

Pattern classification and learning theory (G. Lugosi): A binary classification problem Empirical risk minimization Concentration inequalities Vapnik-Chervonenkis theory Minimax lower bounds Complexity regularization References.- Nonparametric regression estimation (L. Gyorfi, M. Kohler): Regression problem Local averaging estimates Consequences in pattern recognition Definition of (penalized) least squares estimates Consistency of least squares estimates Consistency of penalized least squares estimates Rate of convergence of least squares estimates References.- Universal prediction (N. Cesa-Bianchi): Introduction Potential-based forecasters Convex loss functions Exp-concave loss functions Absolute loss Logarithmic loss Sequentioal pattern classification References.- Learning-theoretic methods in vector quantization (T. Linder): Introduction The fixed-rate quantization problem Consistency of empirical design Finite sample upper bounds Minimax lower bounds Fundamentals of variable-rate quantization The Lagrangian formulation Consistency of Lagrangian empirical design Finite sample bounds in Lagrangian design References.- Distribution and density estimation (L. Devroye, L. Gyorfi): Distribution estimation The density estimation problem The histogram density estimate Choosing Between Two Densities The Minimum Distance Estimate The Kernel Density Estimate Additive Estimates and Data Splitting Bandwidth Selection for Kernel Estimates References.- Programming applied to model identification (M. Sebag): Summary Introduction Artificial Evolution Genetic Programming Genetic Programming with Grammars Discussion and Conclusion References

Nonparametric Regression Estimation

The basic aim of mathematical statistics is to learn a probability law or its characteristics from data.

Minimum kolmogorov distance estimates of parameters and parametrized distributions

Minimum Kolmogorov distance estimates of arbitrary parameters are considered. They are shown to be strongly consistent if the parameter space metric is topologically weaker than the metric induced by the Kolmogorov distance of distributions from the statistical model. If the parameter space metric can be locally uniformly upper-bounded by the induced metric then these estimates are shown to be consistent of ordern−1/2. Similar results are proved for minimum Kolmogorov distance estimates of densities from parametrized families where the consistency is considered in theL1-norm. The presented conditions for the existence, consistency, and consistency of ordern−1/2 are much weaker than those established in the literature for estimates with similar properties. It is shown that these assumptions are satisfied e.g. by all location and scale models with parent distributions different from Dirac, and by all standard exponential models.

On Universal Noiseless Source Coding for Infinite Source Alphabets

We show that there is a universal noiseless source code for the class of all coun-tably infinite memoryless sources for which a fixed given uniquely decodable code has finite expected codeword length. This source code is derived from a class of distribution estimation procedures which are consistent in expected information divergence.

Distribution Estimates Consistent in χ2-Divergence

For a class of histogram based distribution estimators it is proved the consistency in χ2-divergence and expected χ2-divergence. For one of these estimators, introduced formerly by Barron, is also evaluated the rate of consistency in the expected χ2-divergence. These results are stronger than similar results formerly established in the literature for the total variation and Kullbacks discrimination information, which are divergences dominated by the χ2-divergence.