Flemming Topsøe

Some inequalities for information divergence and related measures of discrimination

Inequalities which connect information divergence with other measures of discrimination or distance between probability distributions are used in information theory and its applications to mathematical statistics, ergodic theory, and other scientific fields. We suggest new inequalities of this type, often based on underlying identities. As a consequence, we obtain certain improvements of the well-known Pinsker inequality. Our study depends on two measures of discrimination, called capacitory discrimination and triangular discrimination. The discussion contains references to related research and comparison with other measures of discrimination, e.g., Ali-Silvey-Csiszar (1996, 1966) divergences and, in particular, the Hellinger distance.

Jensen-Shannon divergence and Hilbert space embedding

This paper describes the Jensen-Shannon divergence (JSD) and Hilbert space embedding. With natural definitions making these considerations precise, one finds that the general Jensen-Shannon divergence related to the mixture is the minimum redundancy, which can be achieved by the observer. The set of distributions with the metric /spl radic/JSD can even be embedded isometrically into Hilbert space and the embedding can be identified.

Refinements of Pinsker's inequality

Let V and D denote, respectively, total variation and divergence. We study lower bounds of D with V fixed. The theoretically best (i.e., largest) lower bound determines a function L=L(V), Vajdas (1970) tight lower bound. The main result is an exact parametrization of L. This leads to Taylor polynomials which are lower bounds for L, and thereby to extensions of the classical Pinsker (1960) inequality which has numerous applications, cf. Pinsker and followers.

Maximum entropy versus minimum risk and applications to some classical discrete distributions

The game which can be taken to lie behind the maximum-entropy principle is studied. Refining previous techniques, new theoretical results are obtained. These results are illustrated by concrete examples pertaining to well-known classical models.

Game Theoretical Equilibrium, Maximum Entropy and Minimum Information Discrimination

Games are considered which rely on the concept of a code and which focus on the interplay between the observer and the system being observed. The games lead to specific principles of Game Theoretical Equilibrium. From these, one readily deduces the well known principles of Maximum Entropy and Minimum Information Discrimination. The game theoretical principles apply in certain situations where the classical principles do not.

On Asymptotically Optimal Methods of Prediction and Adaptive Coding for Markov Sources

The problem of predicting a sequence x1, x2, ? generated by a discrete source with unknown statistics is considered. Each letter xt+1 is predicted using information on the word x1x2?xt only. In fact, this problem is a classical problem which has received much attention. Its history can be traced back to Laplace. To estimate the efficiency of a method of prediction, three quantities are considered: the precision as given by the Kullback?Leibler divergence, the memory size of the program needed to implement the method on a computer, and the time required, measured by the number of binary operations needed at each time instant. A method is presented for which the memory size and the average time are close to the minimum. The results can readily be translated to results about adaptive coding.

Unified approach to optimization techniques in Shannon theory

The authors present a unified approach to optimization techniques in Shannon theory. They illustrate their approach with the following examples: code length game where the cost function is zero and there are no side conditions; a game with no side conditions; and a prediction problem where one has to guess the second letter in a sequence of 2 letters from an alphabet.

On the Glivenko-Cantelli Theorem

SummaryVarious generalizations of the classical Glivenko-Cantelli theorem are proved. In particular, we have strived for as general results as possible for theoretical distributions on euclidean spaces, which are absolutely continuous with respect to Lebesgue measure.

Information Theory at the Service of Science

Information theory is becoming more and more important for many fields. This is true for engineering- and technology-based areas but also for more theoretically oriented sciences such as probability and statistics.

Uniformity in convergence of measures

AbstractAssume that a net (σα) of measures converges in some sense to a measureΜ. Then we investigate whether for a given class ℰ of functions, we can conclude that