Andreas Knoblauch

Memory capacities for synaptic and structural plasticity

Neural associative networks with plastic synapses have been proposed as computational models of brain functions and also for applications such as pattern recognition and information retrieval. To guide biological models and optimize technical applications, several definitions of memory capacity have been used to measure the efficiency of associative memory. Here we explain why the currently used performance measures bias the comparison between models and cannot serve as a theoretical benchmark. We introduce fair measures for information-theoretic capacity in associative memory that also provide a theoretical benchmark. In neural networks, two types of manipulating synapses can be discerned: synaptic plasticity, the change in strength of existing synapses, and structural plasticity, the creation and pruning of synapses. One of the new types of memory capacity we introduce permits quantifying how structural plasticity can increase the network efficiency by compressing the network structure, for example, by pruning unused synapses. Specifically, we analyze operating regimes in the Willshaw model in which structural plasticity can compress the network structure and push performance to the theoretical benchmark. The amount C of information stored in each synapse can scale with the logarithm of the network size rather than being constant, as in classical Willshaw and Hopfield nets ( ln 2 0.7). Further, the review contains novel technical material: a capacity analysis of the Willshaw model that rigorously controls for the level of retrieval quality, an analysis for memories with a nonconstant number of active units (where C 1eln 2 0.53), and the analysis of the computational complexity of associative memories with and without network compression.

Pattern separation and synchronization in spiking associative memories and visual areas.

Scene analysis in the mammalian visual system, conceived as a distributed and parallel process, faces the so-called binding problem. As a possible solution, the temporal correlation hypothesis has been suggested and implemented in phase-coding models. We propose an alternative model that reproduces experimental findings of synchronized and desynchronized fast oscillations more closely. This model is based on technical considerations concerning improved pattern separation in associative memories on the one hand, and on known properties of the visual cortex on the other. It consists of two reciprocally connected areas, one corresponding to a peripheral visual area (P), the other a central association area (C). P implements the orientation-selective subsystem of the primary visual cortex, while C was modeled as an associative memory with connections formed by Hebbian learning of all assemblies corresponding to stimulus objects. Spiking neurons including habituation and correlated noise were incorporated as well as realistic synaptic delays. Three learned stimuli were presented simultaneously and correlation analysis was performed on spike recordings. Generally, we found two states of activity: (i) relatively slow and unordered oscillations at about 20-25 Hz, synchronized only within small regions; and (ii) faster and more precise oscillations around 50-60 Hz, synchronized over the whole simulated area. The neuron groups representing one stimulus tended to be simultaneously in either the slow or the fast state. At each particular time, only one assembly was found to be in the fast state. Activation of the three assemblies switched on a time scale of 100 ms. This can be interpreted as self-generated attention switching. On the time scale corresponding to gamma oscillations, cross correlations between local neuron groups were either modulated or flat. Modulated correlograms resulted if the groups coded features corresponding to a common object. Otherwise, the correlograms remained flat. This behavior is in agreement with experimental results, while phase-code models would generally predict modulated correlations also in the case of different objects. Furthermore, we derive a technical version from our biological associative memory model that accomplishes fast pattern separation parallel in O(log2 n) steps for n neurons and sparse coding.

2009 Special Issue: Discrete combinatorial circuits emerging in neural networks: A mechanism for rules of grammar in the human brain?

In neural network research on language, the existence of discrete combinatorial rule representations is commonly denied. Combinatorial capacity of networks and brains is rather attributed to probability mapping and pattern overlay. Here, we demonstrate that networks incorporating relevant features of neuroanatomical connectivity and neuronal function give rise to discrete neuronal circuits that store combinatorial information and exhibit a function similar to elementary rules of grammar. Key properties of these networks are rich auto- and hetero-associative connectivity, availability of sequence detectors similar to those found in a range of animals, and unsupervised Hebbian learning. Input of specific word sequences establishes sequence detectors in the network, and substitutions of words and larger string segments from one syntactic category, occurring in the context of elements of a second syntactic class, lead to binding between them into neuronal assemblies. Critically, these newly formed aggregates of sequence detectors now respond in a discrete generalizing fashion when members of specific substitution classes of string elements are combined with each other. The discrete combinatorial neuronal assemblies (DCNAs) even respond in the same way to learned strings and to word sequences that never appeared in the input but conform to a rule. We also show how combinatorial information interacts with information about functional and anatomical properties of the brain in the emergence of discrete neuronal circuits that may implement rules and discuss the model in the wider context of brain mechanism for syntax and grammar. Implications for the evolution of human language are discussed in closing.

Scene segmentation by spike synchronization in reciprocally connected visual areas. II. Global assemblies and synchronization on larger space and time scales

Abstract. We present further simulation results of the model of two reciprocally connected visual areas proposed in the first paper [Knoblauch and Palm (2002) Biol Cybern 87:151–167]. One area corresponds to the orientation–selective subsystem of the primary visual cortex, the other is modeled as an associative memory representing stimulus objects according to Hebbian learning. We examine the scene-segmentation capability of our model on larger time and space scales, and relate it to experimental findings. Scene segmentation is achieved by attention switching on a time-scale longer than the gamma range. We find that the time-scale can vary depending on habituation parameters in the range of tens to hundreds of milliseconds. The switching process can be related to findings concerning attention and biased competition, and we reproduce experimental poststimulus time histograms (PSTHs) of single neurons under different stimulus and attentional conditions. In a larger variant the model exhibits traveling waves of activity on both slow and fast time-scales, with properties similar to those found in experiments. An apparent weakness of our standard model is the tendency to produce anti-phase correlations for fast activity from the two areas. Increasing the inter-areal delays in our model produces alternations of in-phase and anti-phase oscillations. The experimentally observed in-phase correlations can most naturally be obtained by the involvement of both fast and slow inter-areal connections; e.g., by two axon populations corresponding to fast-conducting myelinated and slow-conducting unmyelinated axons.

Neural associative memory for brain modeling and information retrieval

This work concisely reviews and unifies the analysis of different variants of neural associative networks consisting of binary neurons and synapses (Willshaw model). We compute storage capacity, fault tolerance, and retrieval efficiency and point out problems of the classical Willshaw model such as limited fault tolerance and restriction to logarithmically sparse random patterns. Then we suggest possible solutions employing spiking neurons, compression of the memory structures, and additional cell layers. Finally, we discuss from a technical perspective whether distributed neural associative memories have any practical advantage over localized storage, e.g., in compressed look-up tables.

Scene segmentation by spike synchronization in reciprocally connected visual areas. I. Local effects of cortical feedback.

Abstract. To investigate scene segmentation in the visual system we present a model of two reciprocally connected visual areas using spiking neurons. Area P corresponds to the orientation-selective subsystem of the primary visual cortex, while the central visual area C is modeled as associative memory representing stimulus objects according to Hebbian learning. Without feedback from area C, a single stimulus results in relatively slow and irregular activity, synchronized only for neighboring patches (slow state), while in the complete model activity is faster with an enlarged synchronization range (fast state). When presenting a superposition of several stimulus objects, scene segmentation happens on a time scale of hundreds of milliseconds by alternating epochs of the slow and fast states, where neurons representing the same object are simultaneously in the fast state. Correlation analysis reveals synchronization on different time scales as found in experiments (designated as tower, castle, and hill peaks). On the fast time scale (tower peaks, gamma frequency range), recordings from two sites coding either different or the same object lead to correlograms that are either flat or exhibit oscillatory modulations with a central peak. This is in agreement with experimental findings, whereas standard phase-coding models would predict shifted peaks in the case of different objects.

Combining visual attention, object recognition and associative information processing in a neurobotic system

We have implemented a neurobiologically plausible system on a robot that integrates visual attention, object recognition, language and action processing using a coherent cortex-like architecture based on neural associative memories. This system enables the robot to respond to spoken commands like ”bot show plum” or ”bot put apple to yellow cup”. The scenario for this is a robot close to one or two tables carrying certain kinds of fruit and other simple objects. Tasks such as finding and pointing to certain fruits in a complex visual scene according to spoken or typed commands can be demonstrated. This involves parsing and understanding of simple sentences, relating the nouns to concrete objects sensed by the camera, and coordinating motor output with planning and sensory processing.

Does Spike-Timing-Dependent Synaptic Plasticity Couple or Decouple Neurons Firing in Synchrony?

Spike synchronization is thought to have a constructive role for feature integration, attention, associative learning, and the formation of bidirectionally connected Hebbian cell assemblies. By contrast, theoretical studies on spike-timing-dependent plasticity (STDP) report an inherently decoupling influence of spike synchronization on synaptic connections of coactivated neurons. For example, bidirectional synaptic connections as found in cortical areas could be reproduced only by assuming realistic models of STDP and rate coding. We resolve this conflict by theoretical analysis and simulation of various simple and realistic STDP models that provide a more complete characterization of conditions when STDP leads to either coupling or decoupling of neurons firing in synchrony. In particular, we show that STDP consistently couples synchronized neurons if key model parameters are matched to physiological data: First, synaptic potentiation must be significantly stronger than synaptic depression for small (positive or negative) time lags between presynaptic and postsynaptic spikes. Second, spike synchronization must be sufficiently imprecise, for example, within a time window of 5-10 ms instead of 1 ms. Third, axonal propagation delays should not be much larger than dendritic delays. Under these assumptions synchronized neurons will be strongly coupled leading to a dominance of bidirectional synaptic connections even for simple STDP models and low mean firing rates at the level of spontaneous activity.

Neural associative memory with optimal bayesian learning

Neural associative memories are perceptron-like single-layer networks with fast synaptic learning typically storing discrete associations between pairs of neural activity patterns. Previous work optimized the memory capacity for various models of synaptic learning: linear Hopfield-type rules, the Willshaw model employing binary synapses, or the BCPNN rule of Lansner and Ekeberg, for example. Here I show that all of these previous models are limit cases of a general optimal model where synaptic learning is determined by probabilistic Bayesian considerations. Asymptotically, for large networks and very sparse neuron activity, the Bayesian model becomes identical to an inhibitory implementation of the Willshaw and BCPNN-type models. For less sparse patterns, the Bayesian model becomes identical to Hopfield-type networks employing the covariance rule. For intermediate sparseness or finite networks, the optimal Bayesian learning rule differs from the previous models and can significantly improve memory performance. I also provide a unified analytical framework to determine memory capacity at a given output noise level that links approaches based on mutual information, Hamming distance, and signal-to-noise ratio.

Synaptic plasticity,conduction delays,and inter-areal phase relations of spike activity in a model of reciprocally connected areas

Multi-electrode recordings revealed that fast oscillatory (30 –60 Hz) spike activity is often synchronized with zero phase lag, even for recording sites in distant cortical areas. We show in this simulation study that the dominance of zero-phase correlations is inconsistent with the long conduction delays measured between distant areas: For realistic delays, reciprocally connected neuron populations exhibit anti-phase rather than zero-phase correlations. We show that this inconsistency can be removed by taking into account spike-timing-dependent synaptic plasticity (STDP), as found in experiments (Science 275 (1997) 213). We demonstrate that STDP can weaken fast excitatory feedback and strengthen slower feedback with delays in the range of one oscillation period. This yields stable zero-lag oscillations, even for realistic delays. c � 2003 Elsevier Science B.V. All rights reserved.