Ilya Nemenman

Predictability, Complexity, and Learning

We define predictive information Ipred(T) as the mutual information between the past and the future of a time series. Three qualitatively different behaviors are found in the limit of large observation times T: Ipred(T) can remain finite, grow logarithmically, or grow as a fractional power law. If the time series allows us to learn a model with a finite number of parameters, then Ipred(T) grows logarithmically with a coefficient that counts the dimensionality of the model space. In contrast, power-law growth is associated, for example, with the learning of infinite parameter (or non-parametric) models such as continuous functions with smoothness constraints. There are connections between the predictive information and measures of complexity that have been defined both in learning theory and the analysis of physical systems through statistical mechanics and dynamical systems theory. Furthermore, in the same way that entropy provides the unique measure of available information consistent with some simple and plausible conditions, we argue that the divergent part of Ipred(T) provides the unique measure for the complexity of dynamics underlying a time series. Finally, we discuss how these ideas may be useful in problems in physics, statistics, and biology.

Reverse engineering cellular networks

We describe a computational protocol for the ARACNE algorithm, an information-theoretic method for identifying transcriptional interactions between gene products using microarray expression profile data. Similar to other algorithms, ARACNE predicts potential functional associations among genes, or novel functions for uncharacterized genes, by identifying statistical dependencies between gene products. However, based on biochemical validation, literature searches and DNA binding site enrichment analysis, ARACNE has also proven effective in identifying bona fide transcriptional targets, even in complex mammalian networks. Thus we envision that predictions made by ARACNE, especially when supplemented with prior knowledge or additional data sources, can provide appropriate hypotheses for the further investigation of cellular networks. While the examples in this protocol use only gene expression profile data, the algorithms theoretical basis readily extends to a variety of other high-throughput measurements, such as pathway-specific or genome-wide proteomics, microRNA and metabolomics data. As these data become readily available, we expect that ARACNE might prove increasingly useful in elucidating the underlying interaction models. For a microarray data set containing ∼10,000 probes, reconstructing the network around a single probe completes in several minutes using a desktop computer with a Pentium 4 processor. Reconstructing a genome-wide network generally requires a computational cluster, especially if the recommended bootstrapping procedure is used.

Information transduction capacity of noisy biochemical signaling networks.

Noise limits information transfer through a single signaling pathway in a single cell to just one bit. Molecular noise restricts the ability of an individual cell to resolve input signals of different strengths and gather information about the external environment. Transmitting information through complex signaling networks with redundancies can overcome this limitation. We developed an integrative theoretical and experimental framework, based on the formalism of information theory, to quantitatively predict and measure the amount of information transduced by molecular and cellular networks. Analyzing tumor necrosis factor (TNF) signaling revealed that individual TNF signaling pathways transduce information sufficient for accurate binary decisions, and an upstream bottleneck limits the information gained via multiple integrated pathways. Negative feedback to this bottleneck could both alleviate and enhance its limiting effect, despite decreasing noise. Bottlenecks likewise constrain information attained by networks signaling through multiple genes or cells.

Entropy and information in neural spike trains: Progress on the sampling problem

The major problem in information theoretic analysis of neural responses and other biological data is the reliable estimation of entropy-like quantities from small samples. We apply a recently introduced Bayesian entropy estimator to synthetic data inspired by experiments, and to real experimental spike trains. The estimator performs admirably even very deep in the undersampled regime, where other techniques fail. This opens new possibilities for the information theoretic analysis of experiments, and may be of general interest as an example of learning from limited data.

Genome-wide Identification of Post-translational Modulators of Transcription Factor Activity in Human B-Cells

The ability of a transcription factor (TF) to regulate its targets is modulated by a variety of genetic and epigenetic mechanisms, resulting in highly context-dependent regulatory networks. However, high-throughput methods for the identification of proteins that affect TF activity are still largely unavailable. Here we introduce an algorithm, modulator inference by network dynamics (MINDy), for the genome-wide identification of post-translational modulators of TF activity within a specific cellular context. When used to dissect the regulation of MYC activity in human B lymphocytes, the approach inferred novel modulators of MYC function, which act by distinct mechanisms, including protein turnover, transcription complex formation and selective enzyme recruitment. MINDy is generally applicable to study the post-translational modulation of mammalian TFs in any cellular context. As such it can be used to dissect context-specific signaling pathways and combinatorial transcriptional regulation.

Neural coding of natural stimuli: information at sub-millisecond resolution

Our knowledge of the sensory world is encoded by neurons in sequences of discrete, identical pulses termed action potentials or spikes. There is persistent controversy about the extent to which the precise timing of these spikes is relevant to the function of the brain. We revisit this issue, using the motion – sensitive neurons of the fly visual system as a test case. New experimental methods allow us to deliver more nearly natural visual stimuli, comparable to those which flies encounter in free, acrobatic flight, and new mathematical methods allow us to draw more reliable conclusions about the information content of neural responses even when the set of possible responses is very large. We find that significant amounts of visual information are represented by details of the spike train at millisecond and sub-millisecond precision, even though the sensory input has a correlation time of ~60 ms; different patterns of spike timing represent distinct motion trajectories, and the absolute timing of spikes points to particular features of these trajectories with high precision. Under these naturalistic conditions, the systems information transmission rate still increases with higher photon flux, even though individual photoreceptors are counting more than one million photons per second. Further, exploiting the relatively slow dynamics of the stimulus, the system removes redundancy and so generates a more efficient neural code.

The Berry phase and the pump flux in stochastic chemical kinetics

We study a classical two-state stochastic system in a sea of substrates and products (absorbing states), which can be interpreted as a single Michaelis-Menten catalyzing enzyme or as a channel on a cell surface. We introduce a novel general method and use it to derive the expression for the full counting statistics of transitions among the absorbing states. For the evolution of the system under a periodic perturbation of the kinetic rates, the latter contains a term with a purely geometrical (the Berry phase) interpretation. This term gives rise to a pump current between the absorbing states, which is due entirely to the stochastic nature of the system. We calculate the first two cumulants of this current, and we argue that it is observable experimentally.

Complexity through nonextensivity

The problem of defining and studying complexity of a time series has interested people for years. In the context of dynamical systems, Grassberger has suggested that a slow approach of the entropy to its extensive asymptotic limit is a sign of complexity. We investigate this idea further by information theoretic and statistical mechanics techniques and show that these arguments can be made precise, and that they generalize many previous approaches to complexity, in particular, unifying ideas from the physics literature with ideas from learning and coding theory; there are even connections of this statistical approach to algorithmic or Kolmogorov complexity. Moreover, a set of simple axioms similar to those used by Shannon in his development of information theory allows us to prove that the divergent part of the subextensive component of the entropy is a unique complexity measure. We classify time series by their complexities and demonstrate that beyond the “logarithmic” complexity classes widely anticipated in the literature there are qualitatively more complex, “power-law” classes which deserve more attention.

The simplicity of completion time distributions for common complex biochemical processes

Biochemical processes typically involve huge numbers of individual reversible steps, each with its own dynamical rate constants. For example, kinetic proofreading processes rely upon numerous sequential reactions in order to guarantee the precise construction of specific macromolecules. In this work, we study the transient properties of such systems and fully characterize their first passage (completion) time distributions. In particular, we provide explicit expressions for the mean and the variance of the completion time for a kinetic proofreading process and computational analyses for more complicated biochemical systems. We find that, for a wide range of parameters, as the system size grows, the completion time behavior simplifies: it becomes either deterministic or exponentially distributed, with a very narrow transition between the two regimes. In both regimes, the dynamical complexity of the full system is trivial compared to its apparent structural complexity. Similar simplicity is likely to arise in the dynamics of many complex multistep biochemical processes. In particular, these findings suggest not only that one may not be able to understand individual elementary reactions from macroscopic observations, but also that such an understanding may be unnecessary.

Cellular noise and information transmission

The technological revolution in biological research, and in particular the use of molecular fluorescent labels, has allowed investigation of heterogeneity of cellular responses to stimuli on the single cell level. Computational, theoretical, and synthetic biology advances have allowed predicting and manipulating this heterogeneity with an exquisite precision previously reserved only for physical sciences. Functionally, this cell-to-cell variability can compromise cellular responses to environmental signals, and it can also enlarge the repertoire of possible cellular responses and hence increase the adaptive nature of cellular behaviors. And yet quantification of the functional importance of this response heterogeneity remained elusive. Recently the mathematical language of information theory has been proposed to address this problem. This opinion reviews the recent advances and discusses the broader implications of using information-theoretic tools to characterize heterogeneity of cellular behaviors.