Bálint Takács

ε-mdps: learning in varying environments

In this paper e-MDP-models are introduced and convergence theorems are proven using the generalized MDP framework of Szepesvari and Littman. Using this model family, we show that Q-learning is capable of finding near-optimal policies in varying environments. The potential of this new family of MDP models is illustrated via a reinforcement learning algorithm called event-learning which separates the optimization of decision making from the controller. We show that event-learning augmented by a particular controller, which gives rise to an e-MDP, enables near optimal performance even if considerable and sudden changes may occur in the environment. Illustrations are provided on the two-segment pendulum problem.

Independent subspace analysis on innovations

Independent subspace analysis (ISA) that deals with multi-dimensional independent sources, is a generalization of independent component analysis (ICA). However, all known ISA algorithms may become ineffective when the sources possess temporal structure. The innovation process instead of the original mixtures has been proposed to solve ICA problems with temporal dependencies. Here we show that this strategy can be applied to ISA as well. We demonstrate the idea on a mixture of 3D processes and also on a mixture of facial pictures used as two-dimensional deterministic sources. ISA on innovations was able to find the original subspaces, while plain ISA was not.

Relating priming and repetition suppression

We present a prototype of a recently proposed two stage model of the entorhinal-hippocampal loop. Our aim is to form a general computational model of the sensory neocortex. The model--grounded on pure information theoretic principles--accounts for the most characteristic features of long-term memory (LTM), performs bottom-up novelty detection, and supports noise filtering. Noise filtering can also serve to correct the temporal ordering of information processing. Surprisingly, as we examine the temporal characteristics of the model, the emergent dynamics can be interpreted as perceptual priming, a fundamental type of implicit memory. In the models framework, computational results support the hypothesis of a strong correlation between perceptual priming and repetition suppression and this correlation is a direct consequence of the temporal ordering in forming the LTM. We also argue that our prototype offers a relatively simple and coherent explanation of priming and its relation to a general model of information processing by the brain.

Independent component analysis of temporal sequences forms place cells

Abstract It has been suggested that sensory information processing makes use of a factorial code. It has been shown that the major components of the hippocampal-entorhinal loop can be derived by conjecturing that the task of this loop is forming and encoding independent components (ICs), one type of factorial codes. However, continuously changing environment poses additional requirements on the coding that can be (partially) satisfied by extending the analysis to the temporal domain and performing IC analysis on concatenated inputs of time slices. We use computer simulations to decide whether IC analysis on temporal sequences can produce place fields in labyrinths or not.

Independent component analysis forms place cells in realistic robot simulations

It has been argued that the processing of sensory information in the entorhinal-hippocampal loop involves independent component analysis (ICA) on temporally concatenated inputs. Here, we demonstrate that ICA in a realistic robot simulation on a U-shaped track forms place fields similar to those found in rat experiments in vivo.

Simple conditions for forming triangular grids

We have used simple learning rules to study how firing maps containing triangular grids-as found in in vivo experiments-can be developed by Hebbian means in realistic robotic simulations. We started from typical non-local postrhinal neuronal responses. We found that anti-Hebbian weight pruning can develop triangular grids under certain conditions. Experimental evidences and the present study suggest that within this model, whitening is a bottom-up process, whereas weight pruning and possibly the non-linear extension of whitening to bottom-up information maximization are guided by top-down influences that reorganize entorhinal responses. We connect our model to the computational model of the entorhinal-hippocampal region of Lorincz and Buzsaki. In the joined model, the hippocampus is the origin of response reorganization. The joined model may provide insights for memory reorganization guided by hippocampal supervision.

An algorithm for finding reliably schedulable plans

For interacting agents in time-critical applications, learning whether a subtask can be scheduled reliably is an important issue. The identification of sub-problems of this nature may promote e.g., planning, scheduling and segmenting in Markov decision processes. We define a subtask to be schedulable if its execution time has a small variance. We present an algorithm for finding such subtasks.