Storage capacity of phase-coded patterns in sparse neural networks
aa r X i v : . [ q - b i o . N C ] J un epl draft Storage capacity of phase-coded patterns in sparse neural net-works
S. Scarpetta , , F. Giacco , and A. de Candia , , Dipartimento di Fisica “E. R. Caianiello”, Universit`a di Salerno, Italy INFN, Sezione di Napoli e Gruppo Collegato di Salerno Dipartimento di Scienze Fisiche, Universit`a di Napoli Federico II CNR-SPIN, Unit`a di Napoli
PACS – Biological complexity, neural networks
PACS – Neuroscience,learning and memory
PACS – Neuroscience,neuronal network dynamics
Abstract –We study the storage of multiple phase-coded patterns as stable dynamical attractorsin recurrent neural networks with sparse connectivity. To determine the synaptic strength ofexistent connections and store the phase-coded patterns, we introduce a learning rule inspiredto the spike-timing dependent plasticity (STDP). We find that, after learning, the spontaneousdynamics of the network replay one of the stored dynamical patterns, depending on the networkinitialization. We study the network capacity as a function of topology, and find that a small-world-like topology may be optimal, as a compromise between the high wiring cost of long rangeconnections and the capacity increase.
The capacity to memorize and recall multiple items ofinformation is fundamental to normal cognition. Recentresearches suggest that brain operates as a complex non-linear dynamic system, and synchronous and phase-lockedoscillations may play a crucial role in information process-ing, perception, memory, and sensory computation [1–5].There is increasing experimental evidence that informa-tion encoding may depend on the temporal dynamics be-tween neurons, namely, the specific phase alignment ofspikes relative to rhythmic activity across the neuronalpopulation (as reflected in the local field potential) [5–12].Indeed phase-coding, that exploits the precise temporalrelations between the spikes of neurons, may be an effec-tive general strategy to encode information in the cortex[13–20].The importance of precise timing of the neuron activityis also suggested by the rule that controls the potentia-tion or depression of synaptic strengths, namely the spike-timing dependent plasticity (STDP) [21, 22], based on theprecise order and time interval between the pre- and post-synaptic spikes, on a window of tens of milliseconds. Thiskind of plasticity, with acute sensitivity to temporal or-der, has been demonstrated in various neural circuits overa wide spectrum of species, from insects to xenopus laevis frog, rodents and humans [23, 24].The STDP is strongly asymmetric in the order of arrivalof pre- and post-synaptic spikes, usually determining a po-tentiation in the case of causal order (pre-before-post), anddepression in the reverse order. This temporal asymmetryresults in asymmetric connections between neurons, thatis a crucial ingredient to give rise to dynamical patterns,as opposed to static patterns characteristic of symmetricconnections as in the Hopfield model.Another crucial feature of the network is its connectiv-ity and topology. Whereas all-to-all, or random, wiringis usually assumed in many models, this is possible ina tissue culture involving dozens of neurons, but it be-comes less and less feasible when millions of neurons areinvolved, owing to space and energy supply limitations.The topology of the brain connectivity has been designedby evolutionary goals as a compromise between the abil-ity to achieve complex dynamical functions and physicalconstraints and costs minimizations [25, 26].In the last decade, there has been a growing interestin the study of the topological structure of the brain net-work. There is increasing evidence that the connectionsof neurons in many areas of the nervous system have com-plex topology, such as a small world topology [27], highlyp-1. Scarpetta1,2 F. Giacco1,2 A. de Candia3,4,2connected hubs and modularity [28]. Up to now, the onlynervous system to have been comprehensively mapped ata cellular level is the one of Caenorhabditis elegans, and ithas been found that is has indeed a small world structure[27, 29]. The same property was found for functional andanatomical connectivity, in different animals and areas ofthe brain [28, 30, 31].In this paper, we focus on the functioning of the networkas a dynamical associative memory, that is on the abilityof the network to memorize and recall multiple dynamicalpatterns, where each pattern is characterized not by a setof binary states of the neurons, as in the Hopfield model,but rather by a different set of time shifts (phases) betweenthe periodic activities of the neurons. We study how thecapacity of the network depends on the number of neurons,on the number of connections, and on the topology of thenetwork, that is on the distribution of the connectionsbetween neighboring or distant neurons.We consider a network composed by N neurons, with N ( N −
1) possible (directed) connections J ij . The activityof the neuron is represented by a time-dependent variable σ i ( t ), with σ i ( t ) = − σ i ( t ) = 1 (firingneuron). Note that, in a coarse grained view, the variable σ i ( t ) may as well represent a group of neighboring neuronswith a highly correlated activity.Not all the connections are present in the network. Weput neurons randomly in a three-dimensional box withperiodic boundary conditions, with density equal to one.For each neuron, we consider the sphere centered on it,with a radius R such that the sphere contains the z nearestneurons. Then we connect each neuron to (1 − γ ) z neuronschosen randomly within the sphere of the neighbors, and γz neurons chosen randomly in the whole system. In thisway, we realize a network with a given mean connectivity z , and a given fraction γ of long range vs. short rangeconnections.During the learning phase, we force the network to re-produce a number P of spatio-temporal periodic patterns,given by a specified function σ i ( t ) = f ( ω µ t − φ µi ) , (1)where µ = 1 . . . P is the index of the pattern, ω µ the an-gular velocity, φ µi the phases of the neurons, that encodethe relative times at which neurons start to fire in the pat-tern µ , and the function f ( φ ) is periodic of period 2 π . Weconsider the function f ( φ ) = (cid:26) < φ mod 2 π < π, − π < φ mod 2 π < π, (2)with equal times of silent and firing state. The connection J ij represents the strength of the synapse going from neu-ron j to neuron i , or in the coarse grained view the sumof strengths of synapses going from the group of neurons j to the group of neurons i . While in the Hopfield modelthe pattern to be stored σ µi is static, and the learningrule is the outer product J ij ∝ σ µi σ µj , here the pattern to Fig. 1: The learning window A ( τ ) used in the learning ruleto model STDP, introduced and motivated by [32], A ( τ ) = a p e − τ/T p − a D e − ητ/T p if τ > A ( τ ) = a p e ητ/T D − a D e τ/T D if τ <
0, with a p = γ [1 /T p + η/T D ] − , a D = γ [ η/T p + 1 /T D ] − , T p = 10 . T D = 28 . η = 4, γ = 42. be stored σ µi ( t ) is time dependent, and we formulate thechange in the connections J ij in analogy with the STDP,as δJ ij = t max Z dt t max Z dt ′ σ i ( t ) A ( t − t ′ ) σ j ( t ′ ) , (3)where A ( τ ) is the learning function, and [0 , t max ] is thelearning time interval in which the network is forced toreproduce the pattern µ . We use for the function A ( τ ) theone introduced and motivated by [32], with the parametersthat fit the experimental data of [22] (see Fig. 1). In termsof Fourier components, this is written as δJ µij = t max ∞ X n = −∞ | ˜ f n | ˜ A ( nω µ ) e in ( φ µi − φ µj ) , (4)with ˜ f n = π R π dφ f ( φ ) e inφ , and ˜ A ( ω ) = R dt A ( t ) e iωt .Due to the temporal asymmetry of A ( τ ), the Fouriercomponent ˜ A ( nω µ ) has an imaginary part, and there-fore δJ µij = δJ µji . When we store multiple patterns µ =1 , , . . . , P , the learned weights are the sum of the con-tributions from individual patterns. For the sake of sim-plicity, we consider the same learning time t max and inputfrequency ω µ for all the encoded patterns. Moreover, foreach pattern µ , we extract the phases φ µi randomly anduniformly from the interval [0 , π ).After the learning phase, we initialize the network witha given initial condition σ i (0), and perform a zero temper-ature spontaneous dynamics, σ i ( t + τ ) = sign X j = i J ij σ j ( t ) (5)with τ the unit step of time. Due to the shape of theresponse function sign( x ), if all the connections J ij aremultiplied by the same positive constant, the dynamicsis unchanged. Therefore, the learning time t max and theamplitude of the learning function A ( τ ) are immaterial.p-2torage capacity of phase-coded patterns in sparse neural networksa) b) Fig. 2: a) Plot of the self-sustained dynamics of the fully con-nected network with N = 500 and P = 5. The dynamics ofthe network, after a transient, is periodic of period T = 8 τ .Different colors represent the value of ⌊ t i /T ⌋ mod 4, where t i is time at which the neuron i start to fire (black if neuron isfiring at t = 0). In this case the times t i are highly correlatedwith the phases φ i , which means that the pattern µ = 1 is wellretrieved, and the overlap m ( t ) is large. b) The same networkwith P = 50. In this case the pattern is not retrieved, and theoverlap is of order p /N . In order to measure the similarity between the networkactivity during retrieval mode, and the stored phase-codedpattern µ , we define the overlap m µ ( t ) = 1 N N X i =1 σ i ( t ) e iφ µi . (6)When the retrieval is perfect, σ i ( t ) = f (˜ ωt − φ µi ), with anoutput angular velocity ˜ ω that is in general different fromthe input ω µ . In this case m µ ( t ) = ˜ f e i ˜ ωt , where | ˜ f | ≃ .
64 for the function (2). When the retrieval is not perfect,the modulus | m µ ( t ) | after a transient goes to a constantvalue lower than the maximal one. Finally, if the networkis not able to reproduce the pattern, the overlap becomesafter a transient of order 1 / √ N . The modulus of m µ ( t ) istherefore an order parameter which measure how much thenetwork dynamics match the stored phase-coded pattern µ . Note that the output frequency ˜ ω/ π depends on theinput frequency ω µ / π , and on the degree of asymmetryof the learning function A ( τ ). If ˜ ω is different from ω µ ,and | m µ ( t ) | is high, it means that the phase-coded storedpattern µ is replayed at a frequency different from theone used to store it, i.e. at a different time scale, butwith the same phase relationship. In Fig. 2a we show thedynamics of a network of N = 500 neurons, with all the N ( N −
1) connections activated, and with P = 5 patternsencoded at input frequency ω µ / π = 10 Hz. The networkis initialized with a high overlap with the pattern µ = 1,setting σ i (0) = (cid:26) < φ i < π, − π < φ i < π. (7)Each segment represent a time interval in which the neu-ron i is firing, that is σ i ( t ) = 1, with the neurons ordered a) b) Fig. 3: a) Storage capacity of the network, as a function ofthe fraction of connections z/N and the number of patternsper neuron
P/N , in the region of low connectivity, with N =40000 neurons. Solid (empty) circles represent points wherethe network is (is not) able to retrieve patterns. The networkwith N = 4000 gives the same result (within errors). In thisregion of low z/N the capacity is proportional to the numberof connections. b) Storage capacity in the entire range of z/N ,with N = 4000 neurons. For clarity only a spline separatingthe encodable and the not-encodable region is shown. Inset:comparison between the N = 4000 and the N = 40000 cases fortwo values of z/N . The dashed line in all figures correspondsto P = 0 . z . on the vertical axis by the value of the phase of the firstpattern φ i . In this case the pattern µ = 1 has been re-trieved, and the overlap m ( t ) has modulus | m ( t ) | ≃ . P = 50 patterns encoded. In this case the pattern is notretrieved, and the overlap has modulus | m ( t ) | ≃ . P max of the network as the max-imum number of patterns encodable in the network, andretrievable with an overlap greater than 0 .
2. We have firststudied the capacity of the network as a function of thenumber of connections, in the case of a fraction γ = 1 oflong range connections. As the probability to create a longrange connections is independent on the distance of theneurons, in this case the network is completely random.We therefore consider a network of N neurons, and createa connection from each neuron to a number z of randomlychosen neurons, with 0 < z < N . We then look if the net-work is able to encode P (random) pattern, and retrieveone of them subsequently. We initialize the network withan high overlap with one of the patterns, as in Eq. (7), andsimulate the dynamics in Eq. (5) up to a time t = 2 × in Monte Carlo steps, looking if the overlap with the cho-sen pattern remains higher than 0 .
2. The experiment isrepeated with three different sets of P patterns. In Fig. 3athe result is shown for N = 40000. Solid (empty) circlesrepresent points where the network was (was not) able toretrieve a pattern at least two times out of three. In prac-tice, for all points considered, the network either was al-ways able to retrieve the pattern (three times out of three),or never (zero times out of three). Furthermore, retrievedpatterns had always an overlap greater than 0 .
36, whilep-3. Scarpetta1,2 F. Giacco1,2 A. de Candia3,4,2a) b)
Fig. 4: a) Storage capacity as a function of the fraction of longrange connections γ , for a fixed number of connections per node(dashed line) or for a fixed total cost of the connections (solidline). b) Relative clustering coefficient and path length as afunction of γ . not retrieved ones had an overlap lower than 0 .
08. Re-sults for N = 4000 were practically the same, with a smallnumber of points near the separation between encodableand not encodable region showing some fluctuations (re-trieving the pattern one or two times out of three). Notethat, in the range of values of z/N considered in Fig. 3a,the maximum capacity P max is well described by a linearfunction of the connectivity, P max = 0 . z .In Fig. 3b, we show the results for the entire range of z/N , from zero to one. When z/N is of the order of unity,the number of connections is of order N , so the simula-tion is too expensive for N = 40000. We therefore studysistematically only the network with N = 4000, but com-pare it with the case N = 40000 for two values of z/N ,finding a good agreement. For clarity we do not showthe points simulated, but only a spline separating the en-codable and the not-encodable region. The dashed lineshows the limit of the capacity for low connectivity z/N , P max = 0 . z . It can be seen that, when the connectivitygrows, there is a saturation effect, so that for z ≃ N onefinds P max /N ≃ . P max ≃ . z .To study the effects of short and long range connectiv-ity, we have then analyzed the capacity of the networkin the case of a fraction γ of long range connections, and(1 − γ ) of short range connections. The result is shownin Fig. 4a (dashed line), for N = 40000 and z = 180( z/N = 0 . R ≃ . L ≃
34. The storage ca-pacity, that is the maximum encodable number of patternsuch that the retrieval gives an overlap greater than 0.2,goes from P max ≃ . z for γ = 0 (only local connections)to P max ≃ . z for γ = 1 (random network). This showsthat the number of connections is not the only parameterthat determines the capacity of the network. Long range(random) connections are more effective than short rangeones in encoding patterns.The topology of the biological networks has been shaped by the evolution, as a compromise between the effective-ness in realizing complex tasks, and cost minimizations.While long range connections are more effective than shortrange ones, they are of course more costly. The trade-offbetween these two requirements will produce a networkwith a finite fraction of long range connections. We there-fore introduce a parameter f , that represents the cost of along range connections with respect to a short range one,and we consider a network with (1 − γ ) z local connections,with neurons chosen randomly within those at distancelower than R , and γz/f long range connections, with neu-rons chosen randomly in the whole system. Varying γ , inthis case it is not the connectivity that is constant, butthe total cost of the connections. The storage capacityas a function of γ is shown in Fig. 4a (solid line), for thecase f = 3. The maximum capacity of the network is real-ized with γ ≃ .
5, that corresponds to about 25% of longrange connections over the total of the connections. Tocharacterize the network, we have also calculated for thedifferent values of γ the clustering coefficient C , definedas the probability that two neurons, that are connected toa third neuron, are themselves connected, and the meanpath length λ , defined as the minimum number of connec-tions needed to go from a node to another node, averagedover all pairs of nodes. In Fig.4b, we show the relativeclustering coefficient ( C − C ) / ( C − C ), and path length( λ − λ ) / ( λ − λ ), where the quantities with subscripts 0and 1 refer to the cases γ = 0 and γ = 1 respectively. Notethat the clustering coefficient decreases much more slowlythan the path length, giving rise to a large region wherethe clustering coefficient is not much lower than that forlocal connections only, while the path length is almost aslow as in the random network. The network therefore, inthe region of intermediate γ that corresponds to optimalcapacity, is of a small-world type [27].In this paper we studied the storage and recall of pat-terns in which information is encoded in the phase-basedtiming of firing relative to the cycle. We proposed aSTDP-based learning rule, and we analyzed the its abil-ity to memorize multiple phase-coded patterns, such thatthe spontaneous dynamics of the network selectively givessustained activity which matches one of the stored phase-coded patterns, depending on the initialization of the net-work. Our work generalizes the Hopfield model, to dynam-ical periodic states, characterized by the relative phases ofthe neurons.We have studied the storage capacity for different de-grees of sparseness and topologies of the connections.Changing the proportion γ between short-range and long-range connections, we go from a three-dimensional net-work with only nearest-neighbors connections ( γ = 0) toa random network ( γ = 1). Small but finite values of γ give a “small world” topology, similar to that found inmany areas of nervous system.We find that in the case of only short range connectionsthe capacity is lowest, while in the case of only long rangeconnections, that corresponds to a completely random net-p-4torage capacity of phase-coded patterns in sparse neural networkswork, the capacity is highest. Moreover, a small but finitefraction of long range connection is enough to enhancethe capacity highly, with respect to the short range case.Therefore if the cost of the connections is taken in account,with long range connections more costly than short rangeones, than the optimal capacity will be given by a smallfraction of long range connections, that corresponds to asmall-world topology.This is in agreement with the observation that small-world attributes, with high clustering coefficient and shortpath length, was found across multiple spatial scales ofcortical organization [31, 33]. This property, first foundin C. elegans, is highly conserved over different type ofmeasurement, and across different species, including cat,monkey and humans, for both functional and anatomicalnetworks [28]. REFERENCES[1]
Buzsaki G. and
Draguhn A. , Science , (2004) 1926.[2] Singer W. , Neuron , (1999) 49.[3] Fries N., Nikoli´c D. and
Singer W. , Trends in Neu-rosciences , (2007) 309.[4] Pastalkova E., Itskov V., Amarasingham A. and
Buzski G. , Science , (2008) 1322.[5] Siegel M., Warden M. R. and
Miller E. K. , Proc NatlAcad Sci USA , (2009) 21341.[6] O’Keefe J. and
Recce M. L. , Hippocampus , (1993)317.[7] Huxter J., Burgess N. and
O’Keefe J. , Nature , (2003) 828.[8] Geisler C., Robbe D., Zugaro M., Sirota A. and
Buzsaki G. , Proc Natl Acad Sci USA , (2007) 8149.[9] Perez-Orive J., Mazor O., Turner G. C., Casse-naer S., Wilson R. I. and
Laurent G. , Science , (2002) 359.[10] Montemurro M. A., Rasch M. J., Murayama Y., Lo-gothetis N. K. and
Panzeri S. , Current Biology , (2008) 375.[11] Kayser C., Montemurro M. A., Logothetis N. K. and
Panzeri S. , Neuron , (2009) 597.[12] Panzeri S., Brunel N., Logothetis N. and
KayserC. , Trends in Neurosciences , (2010) 111.[13] Scarpetta S., Zhaoping L. and
Hertz J. , Neural Com-putation , (2002) 2371.[14] Lengyel M., Kwag J., Paulsen O. and
Dayan P. , Na-ture Neuroscience , (2005) 1677.[15] Memmesheimer R. M. and
Timme M. , Phys. Rev. Lett. , (2006) 188101.[16] Yoshioka M., Scarpetta S. and
Marinaro M. , Phys.Rev. E , (2007) 051917.[17] Yoshioka M. , Phys. Rev. Lett. , (2009) 158102.[18] Bush D., Philippides A., Husbands P. and
O’SheaM. , PLoS Comput Biol. , (2010) e1000839.[19] Masquelier T., Hugues E., Deco G. and
Thorpe S. , J. Neurosci. , (2009) 13484.[20] Flight M. H. , Nature Reviews Neuroscience , (2010)834.[21] Markram H., Lubke J., Frotsher M. and
SakmannB. , Science , (1997) 213. [22] Bi G. Q. and
Poo M. M. , J. Neurosci. , (1998) 10464.[23] Caporale N. and
Dan Y. , Annu. Rev. Neurosci. , (2008) 25.[24] Siostrom P. J. , Front. Syn. Neurosc. , (2010) 4.[25] Koch C. and
Laurent G. , Science , (1999) 96.[26] Laughlin S. B. and
Sejnowski T. J. , Science , (2003) 1870.[27] Watts D. J. and
Strogatz S. H. , Nature , (1998)440.[28] Bullmore E. and
Sporns O. , Nat. Rev. Neurosci. , (2009) 186.[29] Latora V. and
Marchiori M. , Eur. Phys. J. B , (2003) 249.[30] Yu S., Huang D., Singer W. and
Nikoli´c D. , CerebralCortex , (2008) 2891.[31] Sporns O. and
Zwi J. D. , Neuroinformatics , (2004)145.[32] Abarbanel H., Huerta R. and
Rabinovich M. I. , ProcNatl Acad Sci USA , (2002) 10132.[33] Sporns O., Chialvo D., Kaiser M. and
Hilgetag C. , Trends in Cognitive Science , (2004) 418.(2004) 418.