Andrea Galluzzi

Retrieval capabilities of hierarchical networks: From dyson to hopfield

Elena Agliari, Adriano Barra, Andrea Galluzzi, Francesco Guerra, Daniele Tantari, and Flavia Tavani Dipartimento di Fisica, Sapienza Università di Roma, P.le A. Moro 2, 00185, Roma, Italy. Dipartimento di Matematica, Sapienza Università di Roma, P.le Aldo Moro 2, 00185, Roma, Italy. Dipartimento SBAI (Ingegneria), Sapienza Università di Roma, Via A. Scarpa 14, 00185, Roma, Italy. (Dated: July 21, 2014)

Hierarchical neural networks perform both serial and parallel processing

In this work we study a Hebbian neural network, where neurons are arranged according to a hierarchical architecture such that their couplings scale with their reciprocal distance. As a full statistical mechanics solution is not yet available, after a streamlined introduction to the state of the art via that route, the problem is consistently approached through signal-to-noise technique and extensive numerical simulations. Focusing on the low-storage regime, where the amount of stored patterns grows at most logarithmical with the system size, we prove that these non-mean-field Hopfield-like networks display a richer phase diagram than their classical counterparts. In particular, these networks are able to perform serial processing (i.e. retrieve one pattern at a time through a complete rearrangement of the whole ensemble of neurons) as well as parallel processing (i.e. retrieve several patterns simultaneously, delegating the management of different patterns to diverse communities that build network). The tune between the two regimes is given by the rate of the coupling decay and by the level of noise affecting the system. The price to pay for those remarkable capabilities lies in a networks capacity smaller than the mean field counterpart, thus yielding a new budget principle: the wider the multitasking capabilities, the lower the network load and vice versa. This may have important implications in our understanding of biological complexity.

Emerging heterogeneities in Italian customs and comparison with nearby countries

In this work we apply techniques and modus operandi typical of Statistical Mechanics to a large dataset about key social quantifiers and compare the resulting behaviors of five European nations, namely France, Germany, Italy, Spain and Switzerland. The social quantifiers considered are i. the evolution of the number of autochthonous marriages (i.e., between two natives) within a given territorial district and ii. the evolution of the number of mixed marriages (i.e., between a native and an immigrant) within a given territorial district. Our investigations are twofold. From a theoretical perspective, we develop novel techniques, complementary to classical methods (e.g., historical series and logistic regression), in order to detect possible collective features underlying the empirical behaviors; from an experimental perspective, we evidence a clear outline for the evolution of the social quantifiers considered. The comparison between experimental results and theoretical predictions is excellent and allows speculating that France, Italy and Spain display a certain degree of internal heterogeneity, that is not found in Germany and Switzerland; such heterogeneity, quite mild in France and in Spain, is not negligible in Italy and highlights quantitative differences in the habits of Northern and Southern regions. These findings may suggest the persistence of two culturally distinct communities, long-term lasting heritages of different and well-established customs. Also, we find qualitative differences between the evolution of autochthonous and of mixed marriages: for the former imitation in decisional mechanisms seems to play a key role (and this results in a square root relation between the number of autochthonous marriages versus the percentage of possible couples inside that country), while for the latter the emerging behavior can be recovered (in most cases) with elementary models with no interactions, suggesting weak imitation patterns between natives and migrants (and this translates in a linear growth for the number of mixed marriages versus the percentage of possible mixed couples in the country). However, the case of mixed marriages displays a more complex phenomenology, where further details (e.g., the provenance and the status of migrants, linguistic barriers, etc.) should also be accounted for.

Mean field bipartite spin models treated with mechanical techniques

Inspired by a continuously increasing interest in modeling and framing complex systems in a thermodynamic rationale, in this paper we continue our investigation in adapting well-known techniques (originally stemmed in fields of physics and mathematics far from the present) for solving for the free energy of mean field spin models in a statistical mechanics scenario. Focusing on the test cases of bipartite spin systems embedded with all the possible interactions (self and reciprocal), we show that both the fully interacting bipartite ferromagnet, as well as the spin glass counterpart, at least at the replica symmetric level, can be solved via the fundamental theorem of calculus, trough an analogy with the Hamilton-Jacobi theory and lastly with a mapping to a Fourier diffusion problem. All these technologies are shown symmetrically for ferromagnets and spin-glasses in full details and contribute as powerful tools in the investigation of complex systems.

Parallel processing in immune networks

In this work, we adopt a statistical-mechanics approach to investigate basic, systemic features exhibited by adaptive immune systems. The lymphocyte network made by B cells and T cells is modeled by a bipartite spin glass, where, following biological prescriptions, links connecting B cells and T cells are sparse. Interestingly, the dilution performed on links is shown to make the system able to orchestrate parallel strategies to fight several pathogens at the same time; this multitasking capability constitutes a remarkable, key property of immune systems as multiple antigens are always present within the host. We also define the stochastic process ruling the temporal evolution of lymphocyte activity and show its relaxation toward an equilibrium measure allowing statistical-mechanics investigations. Analytical results are compared with Monte Carlo simulations and signal-to-noise outcomes showing overall excellent agreement. Finally, within our model, a rationale for the experimentally well-evidenced correlation between lymphocytosis and autoimmunity is achieved; this sheds further light on the systemic features exhibited by immune networks.

Metastable states in the hierarchical Dyson model drive parallel processing in the hierarchical Hopfield network

In this paper, we introduce and investigate the statistical mechanics of hierarchical neural networks. First, we approach these systems ? la Mattis, by thinking of the Dyson model as a single-pattern hierarchical neural network. We also discuss the stability of different retrievable states as predicted by the related self-consistencies obtained both from a mean-field bound and from a bound that bypasses the mean-field limitation. The latter is worked out by properly reabsorbing the magnetization fluctuations related to higher levels of the hierarchy into effective fields for the lower levels. Remarkably, mixing Amit?s ansatz technique for selecting candidate-retrievable states with the interpolation procedure for solving for the free energy of these states, we prove that, due to gauge symmetry, the Dyson model accomplishes both serial and parallel processing. We extend this scenario to multiple stored patterns by implementing the Hebb prescription for learning within the couplings. This results in Hopfield-like networks constrained on a hierarchical topology, for which, by restricting to the low-storage regime where the number of patterns grows at its most logarithmical with the amount of neurons, we prove the existence of the thermodynamic limit for the free energy, and we give an explicit expression of its mean-field bound and of its related improved bound. We studied the resulting self-consistencies for the Mattis magnetizations, which act as order parameters, are studied and the stability of solutions is analyzed to get a picture of the overall retrieval capabilities of the system according to both mean-field and non-mean-field scenarios. Our main finding is that embedding the Hebbian rule on a hierarchical topology allows the network to accomplish both serial and parallel processing. By tuning the level of fast noise affecting it or triggering the decay of the interactions with the distance among neurons, the system may switch from sequential retrieval to multitasking features, and vice versa. However, since these multitasking capabilities are basically due to the vanishing ?dialogue? between spins at long distance, this effective penury of links strongly penalizes the network?s capacity, with results bounded by low storage.

Topological properties of hierarchical networks

Hierarchical networks are attracting a renewal interest for modeling the organization of a number of biological systems and for tackling the complexity of statistical mechanical models beyond mean-field limitations. Here we consider the Dyson hierarchical construction for ferromagnets, neural networks, and spin glasses, recently analyzed from a statistical-mechanics perspective, and we focus on the topological properties of the underlying structures. In particular, we find that such structures are weighted graphs that exhibit a high degree of clustering and of modularity, with a small spectral gap; the robustness of such features with respect to the presence of thermal noise is also studied. These outcomes are then discussed and related to the statistical-mechanics scenario in full consistency. Last, we look at these weighted graphs as Markov chains and we show that in the limit of infinite size, the emergence of ergodicity breakdown for the stochastic process mirrors the emergence of metastabilities in the corresponding statistical mechanical analysis.

Multitasking attractor networks with neuronal threshold noise

We consider the multitasking associative network in the low-storage limit and we study its phase diagram with respect to the noise level T and the degree d of dilution in pattern entries. We find that the system is characterized by a rich variety of stable states, including pure states, parallel retrieval states, hierarchically organized states and symmetric mixtures (remarkably, both even and odd), whose complexity increases as the number of patterns P grows. The analysis is performed both analytically and numerically: Exploiting techniques based on partial differential equations, we are able to get the self-consistencies for the order parameters. Such self-consistency equations are then solved and the solutions are further checked through stability theory to catalog their organizations into the phase diagram, which is outlined at the end. This is a further step towards the understanding of spontaneous parallel processing in associative networks.

A walk in the statistical mechanical formulation of neural networks alternative routes to hebb prescription

Neural networks are nowadays both powerful operational tools (e.g., for pattern recognition, data mining, error correction codes) and complex theoretical models on the focus of scientific investigation. As for the research branch, neural networks are handled and studied by psychologists, neurobiologists, engineers, mathematicians and theoretical physicists. In particular, in theoretical physics, the key instrument for the quantitative analysis of neural networks is statistical mechanics. From this perspective, here, we first review attractor networks: starting from ferromagnets and spin-glass models, we discuss the underlying philosophy and we recover the strand paved by Hopfield, Amit-Gutfreund-Sompolinky. One step forward, we highlight the structural equivalence between Hopfield networks (modeling retrieval) and Boltzmann machines (modeling learning), hence realizing a deep bridge linking two inseparable aspects of biological and robotic spontaneous cognition. As a sideline, in this walk we derive two alternative (with respect to the original Hebb proposal) ways to recover the Hebbian paradigm, stemming from ferromagnets and from spin-glasses, respectively. Further, as these notes are thought of for an Engineering audience, we highlight also the mappings between ferromagnets and operational amplifiers and between antiferromagnets and flip-flops (as neural networks -built by op-amp and flip-flops- are particular spin-glasses and the latter are indeed combinations of ferromagnets and antiferromagnets), hoping that such a bridge plays as a concrete prescription to capture the beauty of robotics from the statistical mechanical perspective.

Ferromagnetic Models for Cooperative Behavior: Revisiting Universality in Complex Phenomena

Ferromagnetic models are harmonic oscillators in statistical mechanics. Beyond their original scope in tackling phase transition and symmetry breaking in theoretical physics, they are nowadays experiencing a renewal applicative interest as they capture the main features of disparate complex phenomena, whose quantitative investigation in the past were forbidden due to data lacking. After a streamlined introduction to these models, suitably embedded on random graphs, aim of the present paper is to show their importance in a plethora of widespread research fields, so to highlight the unifying framework reached by using statistical mechanics as a tool for their investigation. Specifically we will deal with examples stemmed from sociology, chemistry, cybernetics (electronics) and biology (immunology).