Hilbert J. Kappen

Linear theory for control of nonlinear stochastic systems.

We address the role of noise and the issue of efficient computation in stochastic optimal control problems. We consider a class of nonlinear control problems that can be formulated as a path integral and where the noise plays the role of temperature. The path integral displays symmetry breaking and there exists a critical noise value that separates regimes where optimal control yields qualitatively different solutions. The path integral can be computed efficiently by Monte Carlo integration or by a Laplace approximation, and can therefore be used to solve high dimensional stochastic control problems.

Sufficient Conditions for Convergence of the Sum–Product Algorithm

Novel conditions are derived that guarantee convergence of the sum-product algorithm (also known as loopy belief propagation or simply belief propagation (BP)) to a unique fixed point, irrespective of the initial messages, for parallel (synchronous) updates. The computational complexity of the conditions is polynomial in the number of variables. In contrast with previously existing conditions, our results are directly applicable to arbitrary factor graphs (with discrete variables) and are shown to be valid also in the case of factors containing zeros, under some additional conditions. The conditions are compared with existing ones, numerically and, if possible, analytically. For binary variables with pairwise interactions, sufficient conditions are derived that take into account local evidence (i.e., single-variable factors) and the type of pair interactions (attractive or repulsive). It is shown empirically that this bound outperforms existing bounds.

Path integrals and symmetry breaking for optimal control theory

This paper considers linear-quadratic control of a non-linear dynamical system subject to arbitrary cost. I show that for this class of stochastic control problems the non-linear Hamilton–Jacobi–Bellman equation can be transformed into a linear equation. The transformation is similar to the transformation used to relate the classical Hamilton–Jacobi equation to the Schrodinger equation. As a result of the linearity, the usual backward computation can be replaced by a forward diffusion process that can be computed by stochastic integration or by the evaluation of a path integral. It is shown how in the deterministic limit the Pontryagin minimum principle formalism is recovered. The significance of the path integral approach is that it forms the basis for a number of efficient computational methods, such as Monte Carlo sampling, the Laplace approximation and the variational approximation. We show the effectiveness of the first two methods in a number of examples. Examples are given that show the qualitative difference between stochastic and deterministic control and the occurrence of symmetry breaking as a function of the noise.

A generative model for music transcription

In this paper, we present a graphical model for polyphonic music transcription. Our model, formulated as a dynamical Bayesian network, embodies a transparent and computationally tractable approach to this acoustic analysis problem. An advantage of our approach is that it places emphasis on explicitly modeling the sound generation procedure. It provides a clear framework in which both high level (cognitive) prior information on music structure can be coupled with low level (acoustic physical) information in a principled manner to perform the analysis. The model is a special case of the, generally intractable, switching Kalman filter model. Where possible, we derive, exact polynomial time inference procedures, and otherwise efficient approximations. We argue that our generative model based approach is computationally feasible for many music applications and is readily extensible to more general auditory scene analysis scenarios.

Efficient learning in Boltzmann machines using linear response theory

The learning process in Boltzmann machines is computationally very expensive. The computational complexity of the exact algorithm is exponential in the number of neurons. We present a new approximate learning algorithm for Boltzmann machines, based on mean-field theory and the linear response theorem. The computational complexity of the algorithm is cubic in the number of neurons. In the absence of hidden units, we show how the weights can be directly computed from the fixed-point equation of the learning rules. Thus, in this case we do not need to use a gradient descent procedure for the learning process. We show that the solutions of this method are close to the optimal solutions and give a significant improvement when correlations play a significant role. Finally, we apply the method to a pattern completion task and show good performance for networks up to 100 neurons.

Associative memory with dynamic synapses

We have examined a role of dynamic synapses in the stochastic Hopfield-like network behavior. Our results demonstrate an appearance of a novel phase characterized by quick transitions from one memory state to another. The network is able to retrieve memorized patterns corresponding to classical ferromagnetic states but switches between memorized patterns with an intermittent type of behavior. This phenomenon might reflect the flexibility of real neural systems and their readiness to receive and respond to novel and changing external stimuli.

A two-dimensional ensemble coding model for spatial-temporal transformation of saccades in monkey superior colliculus

Fast saccadic eye movements result from the recruitment of a large population of cells in the midbrain superior collicular (SC). The SC transmits its output, which represents a desired vectorial eye displacement in a topographically organized motor map, to horizontal and vertical eye velocity generators in the brainstem. It is generally agreed, that saccade generation is under continuous feedback control. An important problem for models of the saccadic system is, how the spatial representation of the SC output is transformed into the temporal code of brainstem cell population. This problem is even more intriguing, when the possibility is considered of a dynamic feedback loop which includes the SC. Recent experimental findings support such a possibility. In this paper we analyse the properties of a two-dimensional neural network model for the monkey SC, which incorporates a number of realistic features known from neurophysiology and, in addition, is part of the dynamic feedback loop for saccade generation....

An introduction to stochastic control theory, path integrals and reinforcement learning

Control theory is a mathematical description of how to act optimally to gain future rewards. In this paper I give an introduction to deterministic and stochastic control theory and I give an overview of the possible application of control theory to the modeling of animal behavior and learning. I discuss a class of non‐linear stochastic control problems that can be efficiently solved using a path integral or by MC sampling. In this control formalism the central concept of cost‐to‐go becomes a free energy and methods and concepts from statistical physics can be readily applied.

On-line learning processes in artificial neural networks

We study on-line learning processes in artificial neural networks from a general point of view. On-line learning means that a learning step takes place at each presentation of a randomly drawn training pattern. It can be viewed as a stochastic process governed by a continuous-time master equation. On-line learning is necessary if not all training patterns are available all the time. This occurs in many applications when the training patterns are drawn from a time-dependent environmental distribution. Studying learning in a changing environment, we encounter a conflict between the adaptability and the confidence of the networks representation. Minimization of a criterion incorporating both effects yields an algorithm for on-line adaptation of the learning parameter. The inherent noise of on-line learning makes it possible to escape from undesired local minima of the error potential on which the learning rule performs (stochastic) gradient descent. We try to quantify these often made claims by considering the transition times between various minima. We apply our results on the transitions from “twists” in two-dimensional self-organizing maps to perfectly ordered configurations. Finally, we discuss the capabilities of on-line learning for global optimization.

On the properties of the Bethe approximation and loopy belief propagation on binary networks

We analyse the local stability of the high-temperature fixed point of the loopy belief propagation (LBP) algorithm and how this relates to the properties of the Bethe free energy which LBP tries to minimize. We focus on the case of binary networks with pairwise interactions. In particular, we state sufficient conditions for convergence of LBP to a unique fixed point and show that these are sharp for purely ferromagnetic interactions. In contrast, in the purely antiferromagnetic case, the undamped parallel LBP algorithm is suboptimal in the sense that the stability of the fixed point breaks down much earlier than for damped or sequential LBP; we observe that the onset of instability for the latter algorithms is related to the properties of the Bethe free energy. For spin-glass interactions, damping LBP only helps slightly. We estimate analytically the temperature at which the high-temperature LBP fixed point becomes unstable for random graphs with arbitrary degree distributions and random interactions.