Nils Bertschinger

Real-time computation at the edge of chaos in recurrent neural networks

Depending on the connectivity, recurrent networks of simple computational units can show very different types of dynamics, ranging from totally ordered to chaotic. We analyze how the type of dynamics (ordered or chaotic) exhibited by randomly connected networks of threshold gates driven by a time-varying input signal depends on the parameters describing the distribution of the connectivity matrix. In particular, we calculate the critical boundary in parameter space where the transition from ordered to chaotic dynamics takes place. Employing a recently developed framework for analyzing real-time computations, we show that only near the critical boundary can such networks perform complex computations on time series. Hence, this result strongly supports conjectures that dynamical systems that are capable of doing complex computational tasks should operate near the edge of chaos, that is, the transition from ordered to chaotic dynamics.

Quantifying unique information

We propose new measures of shared information, unique information and synergistic information that can be used to decompose the mutual information of a pair of random variables (Y, Z) with a third random variable X. Our measures are motivated by an operational idea of unique information, which suggests that shared information and unique information should depend only on the marginal distributions of the pairs (X, Y) and (X,Z). Although this invariance property has not been studied before, it is satisfied by other proposed measures of shared information. The invariance property does not uniquely determine our new measures, but it implies that the functions that we define are bounds to any other measures satisfying the same invariance property. We study properties of our measures and compare them to other candidate measures.

Autonomy: An information theoretic perspective

We present a tentative proposal for a quantitative measure of autonomy. This is something that, surprisingly, is rarely found in the literature, even though autonomy is considered to be a basic concept in many disciplines, including artificial life. We work in an information theoretic setting for which the distinction between system and environment is the starting point. As a first measure for autonomy, we propose the conditional mutual information between consecutive states of the system conditioned on the history of the environment. This works well when the system cannot influence the environment at all and the environment does not interact synergetically with the system. When, in contrast, the system has full control over its environment, we should instead neglect the environment history and simply take the mutual information between consecutive system states as a measure of autonomy. In the case of mutual interaction between system and environment there remains an ambiguity regarding whether system or environment has caused observed correlations. If the interaction structure of the system is known, we define a causal autonomy measure which allows this ambiguity to be resolved. Synergetic interactions still pose a problem since in this case causation cannot be attributed to the system or the environment alone. Moreover, our analysis reveals some subtle facets of the concept of autonomy, in particular with respect to the seemingly innocent system-environment distinction we took for granted, and raises the issue of the attribution of control, i.e. the responsibility for observed effects. To further explore these issues, we evaluate our autonomy measure for simple automata, an agent moving in space, gliders in the game of life, and the tessellation automaton for autopoiesis of Varela et al. [Varela, F.J., Maturana, H.R., Uribe, R., 1974. Autopoiesis: the organization of living systems, its characterization and a model. BioSystems 5, 187-196].

A Geometric Approach to Complexity

We develop a geometric approach to complexity based on the principle that complexity requires interactions at different scales of description. Complex systems are more than the sum of their parts of any size and not just more than the sum of their elements. Using information geometry, we therefore analyze the decomposition of a system in terms of an interaction hierarchy. In mathematical terms, we present a theory of complexity measures for finite random fields using the geometric framework of hierarchies of exponential families. Within our framework, previously proposed complexity measures find their natural place and gain a new interpretation.

Shared Information—New Insights and Problems in Decomposing Information in Complex Systems

How can the information that a set {X 1,…,X n } of random variables contains about another random variable S be decomposed? To what extent do different subgroups provide the same, i.e. shared or redundant, information, carry unique information or interact for the emergence of synergistic information?

Reconsidering unique information: Towards a multivariate information decomposition

The information that two random variables Y, Z contain about a third random variable X can have aspects of shared information (contained in both Y and Z), of complementary information (only available from (Y, Z) together) and of unique information (contained exclusively in either Y or Z). Here, we study measures SĨ of shared, UĨ unique and CĨ complementary information introduced by Bertschinger et al. [1] which are motivated from a decision theoretic perspective. We find that in most cases the intuitive rule that more variables contain more information applies, with the exception that SĨ and CĨ information are not monotone in the target variable X. Additionally, we show that it is not possible to extend the bivariate information decomposition into SĨ, UĨ and CĨ to a non-negative decomposition on the partial information lattice of Williams and Beer [2]. Nevertheless, the quantities UĨ, SĨ and CĨ have a well-defined interpretation, even in the multivariate setting.

COMPARISON BETWEEN DIFFERENT METHODS OF LEVEL IDENTIFICATION

Levels of a complex system are characterized by the fact that they admit a closed functional description in terms of concepts and quantities intrinsic to that level. Several ideas have come up so far in order to make the notion of a closed description precise. In this paper, we present four of these approaches and investigate their mutual relationships. Our study is restricted to the case of discrete dynamical systems, where the different levels are linked by a coarse-graining of variables and states of the system.

Quantifying structure in networks

Abstract.nWe investigate exponential families of random graph distributions asna framework for systematic quantification of structure innnetworks. In this paper we restrict ourselves to undirectednunlabeled graphs. For these graphs, the counts of subgraphs with nonmore than k links are a sufficient statistics for the exponentialnfamilies of graphs with interactions between at most k links. Innthis framework we investigate the dependencies between severalnobservables commonly used to quantify structure in networks, such asnthe degree distribution, cluster and assortativity coefficients.n

On Extractable Shared Information

We consider the problem of quantifying the information shared by a pair of random variables X 1 , X 2 about another variable S. We propose a new measure of shared information, called extractable shared information, that is left monotonic; that is, the information shared about S is bounded from below by the information shared about f ( S ) for any function f. We show that our measure leads to a new nonnegative decomposition of the mutual information I ( S ; X 1 X 2 ) into shared, complementary and unique components. We study properties of this decomposition and show that a left monotonic shared information is not compatible with a Blackwell interpretation of unique information. We also discuss whether it is possible to have a decomposition in which both shared and unique information are left monotonic.

Closure measures for coarse-graining of the tent map.

We quantify the relationship between the dynamics of a time-discrete dynamical system, the tent map T and its iterations T(m), and the induced dynamics at a symbolical level in information theoretical terms. The symbol dynamics, given by a binary string s of length m, is obtained by choosing a partition point [Formula: see text] and lumping together the points [Formula: see text] s.t. T(i)(x) concurs with the i - 1th digit of s-i.e., we apply a so called threshold crossing technique. Interpreting the original dynamics and the symbolic one as different levels, this allows us to quantitatively evaluate and compare various closure measures that have been proposed for identifying emergent macro-levels of a dynamical system. In particular, we can see how these measures depend on the choice of the partition point α. As main benefit of this new information theoretical approach, we get all Markov partitions with full support of the time-discrete dynamical system induced by the tent map. Furthermore, we could derive an example of a Markovian symbol dynamics whose underlying partition is not Markovian at all, and even a whole hierarchy of Markovian symbol dynamics.