Giovanni Pistone

Algebraic Statistics : Computational Commutative Algebra in Statistics

INTRODUCTION History and Motivation Overview Computer Algebra Summary ALGEBRAIC MODELS Models Polynomials and Polynomial Ideals Term-Orderings Division Algorithm All Ideals Are Finitely Generated Varieties and Equations Grobner Bases Properties of Grobner Basis Elimination Theory Polynomial Functions and Quotients by Ideals Hilbert Function Further Topics THE DIRECT THEORY Designs and Design Ideals Computing the Grobner basis of a design Operations with Designs Examples Span of a Design Models and Identifiability Quotients Examples The Fan of an Experimental Design Subsets and Sequential Algorithms Regression Analysis Other Topics TWO-LEVEL DESIGNS. APPLICATION IN LOGIC AND RELIABILITY The binary case: Boolean Representations Reliability: Coherent Systems are Minimal Fan Designs Two Level Factorial Design: Contrasts and Orthogonality PROBABILITY AND STATISTICS Random Variables on a Finite Support Moments Probability Algebraic Representation of Exponentials Generating Functions Generating Functions and Exponential Models Examples and Further Applications Statistical Modelling Likelihoods and Sufficient Statistics A Ring of Random Variables Score Function and Information

Classification of two-level factorial fractions

Abstract The problem of finding a fraction of a two-level factorial design with specific properties is usually solved within special classes, such as regular or Plackett–Burman designs. We show that each fraction of a two-level factorial design is characterized by the ANOVA representation of its polynomial indicator function. In particular, such a representation can be used to present the problem of finding a fraction with a given orthogonality structure as the set of solutions of a system of algebraic equations. Regularity, resolution, projectivity, absence of coincidence defects can be discussed in this framework. The system of algebraic equations involved can be solved, at least in principle, using Computer Algebra softwares, such as Maple and CoCoA. As tutorial examples, all non-trivial orthogonal fractions of a 24 and of a 25 design are computed and classified.

κ-exponential models from the geometrical viewpoint

We discuss the use of Kaniadakis’ κ-exponential in the construction of a statistical manifold modelled on Lebesgue spaces of real random variables. Some algebraic features of the deformed exponential models are considered. A chart is defined for each strictly positive densities; every other strictly positive density in a suitable neighborhood of the reference probability is represented by the centered lnκ likelihood.

Towards the geometry of estimation of distribution algorithms based on the exponential family

In this paper we present a geometrical framework for the analysis of Estimation of Distribution Algorithms (EDAs) based on the exponential family. From a theoretical point of view, an EDA can be modeled as a sequence of densities in a statistical model that converges towards distributions with reduced support. Under this framework, at each iteration the empirical mean of the fitness function decreases in probability, until convergence of the population. This is the context of stochastic relaxation, i.e., the idea of looking for the minima of a function by minimizing its expected value over a set of probability densities. Our main interest is in the study of the gradient of the expected value of the function to be minimized, and in particular on how its landscape changes according to the fitness function and the statistical model used in the relaxation. After introducing some properties of the exponential family, such as the description of its topological closure and of its tangent space, we provide a characterization of the stationary points of the relaxed problem, together with a study of the minimizing sequences with reduced support. The analysis developed in the paper aims to provide a theoretical understanding of the behavior of EDAs, and in particular their ability to converge to the global minimum of the fitness function. The theoretical results of this paper, beside providing a formal framework for the analysis of EDAs, lead to the definition of a new class algorithms for binary functions optimization based on Stochastic Natural Gradient Descent (SNGD), where the estimation of the parameters of the distribution is replaced by the direct update of the model parameters by estimating the natural gradient of the expected value of the fitness function.

Nonparametric Information Geometry

The differential-geometric structure of the set of positive densities on a given measure space has raised the interest of many mathematicians after the discovery by C.R. Rao of the geometric meaning of the Fisher information. Most of the research is focused on parametric statistical models. In series of papers by author and coworkers a particular version of the nonparametric case has been discussed. It consists of a minimalistic structure modeled according the theory of exponential families: given a reference density other densities are represented by the centered log likelihood which is an element of an Orlicz space. This mappings give a system of charts of a Banach manifold. It has been observed that, while the construction is natural, the practical applicability is limited by the technical difficulty to deal with such a class of Banach spaces. It has been suggested recently to replace the exponential function with other functions with similar behavior but polynomial growth at infinity in order to obtain more tractable Banach spaces, e.g. Hilbert spaces. We give first a review of our theory with special emphasis on the specific issues of the infinite dimensional setting. In a second part we discuss two specific topics, differential equations and the metric connection. The position of this line of research with respect to other approaches is briefly discussed.

Information Geometry Formalism for the Spatially Homogeneous Boltzmann Equation

Information Geometry generalizes to infinite dimension by modeling the tangent space of the relevant manifold of probability densities with exponential Orlicz spaces. We review here several properties of the exponential manifold on a suitable set Ɛ of mutually absolutely continuous densities. We study in particular the fine properties of the Kullback-Liebler divergence in this context. We also show that this setting is well-suited for the study of the spatially homogeneous Boltzmann equation if Ɛ is a set of positive densities with finite relative entropy with respect to the Maxwell density. More precisely, we analyze the Boltzmann operator in the geometric setting from the point of its Maxwell’s weak form as a composition of elementary operations in the exponential manifold, namely tensor product, conditioning, marginalization and we prove in a geometric way the basic facts, i.e., the H-theorem. We also illustrate the robustness of our method by discussing, besides the Kullback-Leibler divergence, also the property of Hyvarinen divergence. This requires us to generalize our approach to Orlicz–Sobolev spaces to include derivatives.

Natural gradient, fitness modelling and model selection: A unifying perspective

The geometric framework based on Stochastic Relaxation allows to describe from a common perspective different model-based optimization algorithms that make use of statistical models to guide the search for the optimum. In this paper Stochastic Relaxation is used to provide theoretical results on Estimation of Distribution Algorithms (EDAs). By the use of Stochastic Relaxation we show how the estimation of the fitness model by least squares linear regression corresponds to the estimation of the natural gradient. This equivalence allows to simultaneously perform model selection and robust estimation of the natural gradient. Finally, we interpet Linear Programming relaxation as an example of Stochastic Relaxation, with respect to the regular gradient.

Examples of the Application of Nonparametric Information Geometry to Statistical Physics

We review a nonparametric version of Amari’s information geometry in which the set of positive probability densities on a given sample space is endowed with an atlas of charts to form a differentiable manifold modeled on Orlicz Banach spaces. This nonparametric setting is used to discuss the setting of typical problems in machine learning and statistical physics, such as black-box optimization, Kullback-Leibler divergence, Boltzmann-Gibbs entropy and the Boltzmann equation.

Confounding revisited with commutative computational algebra

Abstract We use computational commutative algebra to discuss and compute confounding relations for general, e.g. non-regular, fractions of a factorial design. Our method is based on the algebraic description of the design as the set of solutions of a system of polynomial equations. Grobner bases of polynomial ideals are used as computational tools. Symbolic softwares are used to derive confounding relations as normal forms of the interaction terms with respect to a given term ordering.

Stochastic Natural Gradient Descent by estimation of empirical covariances

Stochastic relaxation aims at finding the minimum of a fitness function by identifying a proper sequence of distributions, in a given model, that minimize the expected value of the fitness function. Different algorithms fit this framework, and they differ according to the policy they implement to identify the next distribution in the model. In this paper we present two algorithms, in the stochastic relaxation framework, for the optimization of real-valued functions defined over binary variables: Stochastic Gradient Descent (SGD) and Stochastic Natural Gradient Descent (SNDG). These algorithms use a stochastic model to sample from as it happens for Estimation of Distribution Algorithms (EDAs), but the estimation of the model from the population is substituted by the direct update of model parameter through stochastic gradient descent. The two algorithms, SGD and SNDG, both use statistical models in the exponential family, but they differ in the use of the natural gradient, first proposed in the literature by Amari, in the context of Information Geometry. Due to the properties of the exponential family, both gradient and natural gradient can be evaluated in terms of covariances between the fitness function and the sufficient statistics of the exponential family. As the computation of the exact gradient is unfeasible, we approximate the gradient by evaluating empirical covariances. We test the performance of our algorithm over different standard benchmarks, and we compare the results with other well-known meta-heuristics in the framework of EDAs.