João Sacramento

Neural networks letter: Tree-like hierarchical associative memory structures

In this letter we explore an alternative structural representation for Steinbuch-type binary associative memories. These networks offer very generous storage capacities (both asymptotic and finite) at the expense of sparse coding. However, the original retrieval prescription performs a complete search on a fully-connected network, whereas only a small fraction of units will eventually contain desired results due to the sparse coding requirement. Instead of modelling the network as a single layer of neurons we suggest a hierarchical organization where the information content of each memory is a successive approximation of one another. With such a structure it is possible to enhance retrieval performance using a progressively deepening procedure. To backup our intuition we provide collected experimental evidence alongside comments on eventual biological plausibility.

Energy Efficient Sparse Connectivity from Imbalanced Synaptic Plasticity Rules

It is believed that energy efficiency is an important constraint in brain evolution. As synaptic transmission dominates energy consumption, energy can be saved by ensuring that only a few synapses are active. It is therefore likely that the formation of sparse codes and sparse connectivity are fundamental objectives of synaptic plasticity. In this work we study how sparse connectivity can result from a synaptic learning rule of excitatory synapses. Information is maximised when potentiation and depression are balanced according to the mean presynaptic activity level and the resulting fraction of zero-weight synapses is around 50%. However, an imbalance towards depression increases the fraction of zero-weight synapses without significantly affecting performance. We show that imbalanced plasticity corresponds to imposing a regularising constraint on the L 1-norm of the synaptic weight vector, a procedure that is well-known to induce sparseness. Imbalanced plasticity is biophysically plausible and leads to more efficient synaptic configurations than a previously suggested approach that prunes synapses after learning. Our framework gives a novel interpretation to the high fraction of silent synapses found in brain regions like the cerebellum.

Taxonomical Associative Memory

Assigning categories to objects allows the mind to code experience by concepts, thus easing the burden in perceptual, storage, and reasoning processes. Moreover, maximal efficiency of cognitive resources is attained with categories that best mirror the structure of the perceived world. In this work, we will explore how taxonomies could be represented in the brain, and their application in learning and recall. In a recent work, Sacramento and Wichert (in Neural Netw 24(2):143–147, 2011) proposed a hierarchical arrangement of compressed associative networks, improving retrieval time by allowing irrelevant neurons to be pruned early. We present an extension to this model where superordinate concepts are encoded in these compressed networks. Memory traces are stored in an uncompressed network, and each additional network codes for a taxonomical rank. Retrieval is progressive, presenting increasingly specific superordinate concepts. The semantic and technical aspects of the model are investigated in two studies: wine classification and random correlated data.

Regarding the temporal requirements of a hierarchical Willshaw network

In a recent communication, Sacramento and Wichert (2011) proposed a hierarchical retrieval prescription for Willshaw-type associative networks. Through simulation it was shown that one could make use of low resolution descriptor patterns to decrease the total time requirements of recalling a learnt association. However, such a method introduced a dependence on a set of new parameters which define the structure of the hierarchy. In this work we compute the expected retrieval time for the random neural activity regime which maximises the capacity of the Willshaw model and we study the task of finding the optimal hierarchy parametrisation with respect to the derived temporal expectation. Still in regard to this performance measure, we investigate some asymptotic properties of the algorithm.

Binary Willshaw learning yields high synaptic capacity for long-term familiarity memory

In this study, we investigate from a computational perspective the efficiency of the Willshaw synaptic update rule in the context of familiarity discrimination, a binary-answer, memory-related task that has been linked through psychophysical experiments with modified neural activity patterns in the prefrontal and perirhinal cortex regions. Our motivation for recovering this well-known learning prescription is two-fold: first, the switch-like nature of the induced synaptic bonds, as there is evidence that biological synaptic transitions might occur in a discrete stepwise fashion. Second, the possibility that in the mammalian brain, unused, silent synapses might be pruned in the long-term. Besides the usual pattern and network capacities, we calculate the synaptic capacity of the model, a recently proposed measure where only the functional subset of synapses is taken into account. We find that in terms of network capacity, Willshaw learning is strongly affected by the pattern coding rates, which have to be kept fixed and very low at any time to achieve a non-zero capacity in the large network limit. The information carried per functional synapse, however, diverges and is comparable to that of the pattern association case, even for more realistic moderately low activity levels that are a function of network size.

Backward reasoning the formation rules.

Synaptic plasticity during learning is as fundamental as it is hard to study. The underlying synaptic plasticity rule has now been inferred using only the firing rate statistics of visual neurons in monkeys before and after learning.

Computational roles of plastic probabilistic synapses

The probabilistic nature of synaptic transmission has remained enigmatic. However, recent developments have started to shed light on why the brain may rely on probabilistic synapses. Here, we start out by reviewing experimental evidence on the specificity and plasticity of synaptic response statistics. Next, we overview different computational perspectives on the function of plastic probabilistic synapses for constrained, statistical and deep learning. We highlight that all of these views require some form of optimisation of probabilistic synapses, which has recently gained support from theoretical analysis of long-term synaptic plasticity experiments. Finally, we contrast these different computational views and propose avenues for future research. Overall, we argue that the time is ripe for a better understanding of the computational functions of probabilistic synapses.

Sensory representation of an auditory cued tactile stimulus in the posterior parietal cortex of the mouse

Sensory association cortices receive diverse inputs with their role in representing and integrating multi-sensory content remaining unclear. Here we examined the neuronal correlates of an auditory-tactile stimulus sequence in the posterior parietal cortex (PPC) using 2-photon calcium imaging in awake mice. We find that neuronal subpopulations in layer 2/3 of PPC reliably represent texture-touch events, in addition to auditory cues that presage the incoming tactile stimulus. Notably, altering the flow of sensory events through omission of the cued texture touch elicited large responses in a subset of neurons hardly responsive to or even inhibited by the tactile stimuli. Hence, PPC neurons were able to discriminate not only tactile stimulus features (i.e., texture graininess) but also between the presence and omission of the texture stimulus. Whereas some of the neurons responsive to texture omission were driven by looming-like auditory sounds others became recruited only with tactile sensory experience. These findings indicate that layer 2/3 neuronal populations in PPC potentially encode correlates of expectancy in addition to auditory and tactile stimuli.