Johannes Rauh

Quantifying unique information

We propose new measures of shared information, unique information and synergistic information that can be used to decompose the mutual information of a pair of random variables (Y, Z) with a third random variable X. Our measures are motivated by an operational idea of unique information, which suggests that shared information and unique information should depend only on the marginal distributions of the pairs (X, Y) and (X,Z). Although this invariance property has not been studied before, it is satisfied by other proposed measures of shared information. The invariance property does not uniquely determine our new measures, but it implies that the functions that we define are bounds to any other measures satisfying the same invariance property. We study properties of our measures and compare them to other candidate measures.

Information Decomposition and Synergy

Recently, a series of papers addressed the problem of decomposing the information of two random variables into shared information, unique information and synergistic information. Several measures were proposed, although still no consensus has been reached. Here, we compare these proposals with an older approach to define synergistic information based on the projections on exponential families containing only up to k-th order interactions. We show that these measures are not compatible with a decomposition into unique, shared and synergistic information if one requires that all terms are always non-negative (local positivity). We illustrate the difference between the two measures for multivariate Gaussians.

Shared Information—New Insights and Problems in Decomposing Information in Complex Systems

How can the information that a set {X 1,…,X n } of random variables contains about another random variable S be decomposed? To what extent do different subgroups provide the same, i.e. shared or redundant, information, carry unique information or interact for the emergence of synergistic information?

Reconsidering unique information: Towards a multivariate information decomposition

The information that two random variables Y, Z contain about a third random variable X can have aspects of shared information (contained in both Y and Z), of complementary information (only available from (Y, Z) together) and of unique information (contained exclusively in either Y or Z). Here, we study measures SĨ of shared, UĨ unique and CĨ complementary information introduced by Bertschinger et al. [1] which are motivated from a decision theoretic perspective. We find that in most cases the intuitive rule that more variables contain more information applies, with the exception that SĨ and CĨ information are not monotone in the target variable X. Additionally, we show that it is not possible to extend the bivariate information decomposition into SĨ, UĨ and CĨ to a non-negative decomposition on the partial information lattice of Williams and Beer [2]. Nevertheless, the quantities UĨ, SĨ and CĨ have a well-defined interpretation, even in the multivariate setting.

Support sets in exponential families and oriented matroid theory

The closure of a discrete exponential family is described by a finite set of equations corresponding to the circuits of an underlying oriented matroid. These equations are similar to the equations used in algebraic statistics, although they need not be polynomial in the general case. This description allows for a combinatorial study of the possible support sets in the closure of an exponential family. If two exponential families induce the same oriented matroid, then their closures have the same support sets. Furthermore, the positive cocircuits give a parameterization of the closure of the exponential family.

Selection Criteria for Neuromanifolds of Stochastic Dynamics

We present ways of defining neuromanifolds – models of stochastic matrices – that are compatible with the maximization of an objective function such as the expected reward in reinforcement learning theory. Our approach is based on information geometry and aims to reduce the number of model parameters with the hope to improve gradient learning processes.

Finding the Maximizers of the Information Divergence From an Exponential Family

This paper investigates maximizers of the information divergence from an exponential family ε. It is shown that the rI -projection of a maximizer P to ε is a convex combination of P and a probability measure P- with disjoint support and the same value of the sufficient statistics <i>A</i>. This observation can be used to transform the original problem of maximizing D(·∥ε) over the set of all probability measures into the maximization of a function D̅_r over a convex subset of ker A. The global maximizers of both problems correspond to each other. Furthermore, finding all local maximizers of D̅_r yields all local maximizers of D(·∥E). This paper also proposes two algorithms to find the maximizers of D̅_r and applies them to two examples, where the maximizers of D(·∥ε) were not known before.

Generalized binomial edge ideals

This paper studies a class of binomial ideals associated to graphs with finite vertex sets. They generalize the binomial edge ideals, and they arise in the study of conditional independence ideals. A Grobner basis can be computed by studying paths in the graph. Since these Grobner bases are square-free, generalized binomial edge ideals are radical. To find the primary decomposition a combinatorial problem involving the connected components of subgraphs has to be solved. The irreducible components of the solution variety are all rational.

On Extractable Shared Information

We consider the problem of quantifying the information shared by a pair of random variables X 1 , X 2 about another variable S. We propose a new measure of shared information, called extractable shared information, that is left monotonic; that is, the information shared about S is bounded from below by the information shared about f ( S ) for any function f. We show that our measure leads to a new nonnegative decomposition of the mutual information I ( S ; X 1 X 2 ) into shared, complementary and unique components. We study properties of this decomposition and show that a left monotonic shared information is not compatible with a Blackwell interpretation of unique information. We also discuss whether it is possible to have a decomposition in which both shared and unique information are left monotonic.

POSITIVE MARGINS AND PRIMARY DECOMPOSITION

We study random walks on contingency tables with fixed marginals, cor- responding to a (log-linear) hierarchical model. If the set of allowed moves is not a Markov basis, then there exist tables with the same marginals that are not connected. We study linear conditions on the values of the marginals that ensure that all tables in a given fiber are connected. We show that many graphical models have the posi- tive margins property, which says that all fibers with strictly positive marginals are connected by the quadratic moves that correspond to conditional independence state- ments. The property persists under natural operations such as gluing along cliques, but we also construct examples of graphical models not enjoying this property. Our analysis of the positive margins property depends on computing the primary decomposition of the associated conditional independence ideal. The main technical results of the paper are primary decompositions of the conditional independence ideals of graphical models of the N-cycle and the complete bipartite graph K2,N 2, with various restrictions on the size of the nodes.