E Gardner

The space of interactions in neural network models

The typical fraction of the space of interactions between each pair of N Ising spins which solve the problem of storing a given set of p random patterns as N-bit spin configurations is considered. The volume is calculated explicitly as a function of the storage ratio, alpha =p/N, of the value kappa (>0) of the product of the spin and the magnetic field at each site and of the magnetisation, m. Here m may vary between 0 (no correlation) and 1 (completely correlated). The capacity increases with the correlation between patterns from alpha =2 for correlated patterns with kappa =0 and tends to infinity as m tends to 1. The calculations use a saddle-point method and the order parameters at the saddle point are assumed to be replica symmetric. This solution is shown to be locally stable. A local iterative learning algorithm for updating the interactions is given which will converge to a solution of given kappa provided such solutions exist.

An Exactly Solvable Asymmetric Neural Network Model

We consider a diluted and nonsymmetric version of the Little-Hopfield model which can be solved exactly. We obtain the analytic expression of the evolution of one configuration having a finite overlap on one stored pattern. We show that even when the system remembers, two different configurations which remain close to the same pattern never become identical. Lastly, we show that when two stored patterns are correlated, there exists a regime for which the system remembers these patterns without being able to distinguish them.

Maximum Storage Capacity in Neural Networks

The upper storage capacity of a neural network for patterns of fixed magnetization m is calculated. The optimal capacity increases with the correlation m2 between the patterns.

Optimal storage properties of neural network models

The authors calculate the number, p= alpha N of random N-bit patterns that an optimal neural network can store allowing a given fraction f of bit errors and with the condition that each right bit is stabilised by a local field at least equal to a parameter K. For each value of alpha and K, there is a minimum fraction fmin of wrong bits. They find a critical line, alpha c(K) with alpha c(0)=2. The minimum fraction of wrong bits vanishes for alpha alpha c(K). The calculations are done using a saddle-point method and the order parameters at the saddle point are assumed to be replica symmetric. This solution is locally stable in a finite region of the K, alpha plane including the line, alpha c(K) but there is a line above which the solution becomes unstable and replica symmetry must be broken.

Structure of metastable states in the Hopfield model

An upper bound for the number of metastable states in the Hopfield model is calculated as a function of the Hamming fraction from an input pattern. For all finite values of alpha , the ratio of number of patterns to nodes, the hamming fraction from the input pattern to the nearest metastable state is infinite. When alpha <0.113, the bound also implies that there is a gap between a set of states close to the input pattern and another set centred around the Hamming fraction 0.5 from it.

Solution of the generalised random energy model

The generalised random energy model (GREM) is a spin-glass model which can be solved exactly. One can impose arbitrary pair correlations between the energies of configurations. For several examples (the Sherrington-Kirkpatrick model, the p spin-glass model, the Potts glass, spin-glass models on finite-dimensional lattices) the authors calculate the pair correlation between energies and solve the corresponding GREM. In all cases, the free energy of the GREM corresponding to a spin-glass model on a given lattice, has a simple expression in terms of the specific heat of the pure ferromagnetic model on the same lattice. Lastly they compare the correlations between three energy levels in the GREM and in spin-glass models.

Multiconnected neural network models

A generalisation of the Hopfield model which includes interactions between p()2) Ising spins is considered. The exact storage capacity behaves as Np-1/2(p-1)! ln N when the number of nodes, N, is large. In the limit p to infinity , the thermodynamics of the model can be solved exactly without using the replica method; at zero temperature, a solution which is completely correlated with the input pattern exists for alpha < alpha c where alpha c to infinity as p to infinity and this solution has lower energy than the spin-glass solution if alpha < alpha 1=1/4 ln 2 where the number of patterns n=(2 alpha /p!)Np-1. For finite values of p, the correlation with the input pattern is not complete; for p=3 and 4, approximate values of alpha c and alpha 1 are obtained and for p to infinity the replica symmetric approximation gives alpha c approximately p/4 ln p.

Optimal basins of attraction in randomly sparse neural network models

The size of the basin of attraction for randomly sparse neural networks with optimal interactions is calculated. For all values of the storage ratio, alpha =p/C<2, where p is the number of random uncorrelated patterns and C is the connectivity, the basin of attraction is finite, while for alpha <0.42, the basin of attraction is (almost) 100%.

Training with noise and the storage of correlated patterns in a neural network model

Local iterative learning algorithms for the interactions between Ising spins in neural network models are discussed. They converge to solutions with basins of attraction whose shape is determined by the noise in the training data, provided such solutions exist. The training is applied both to the storage of random patterns and to a model for the storage of correlated words. The existence of correlations increases the storage capacity of a given network beyond that for random patterns. The model can be modified to store cycles of patterns and in particular is applied to the storage of continuous items of English text.

The probability distribution of the partition function of the random energy model

The authors give the expression for both integer and non-integer moments of the partition function Z of the random energy model. In the thermodynamic limit, they find that the probability distribution P(Z) can be decomposed into two parts. For log Z-(log Z) finite, the distribution is independent of N, the size of the system, whereas for log Z-(log Z) positive and of order N, the distribution is Gaussian. These two parts match in the region 1<<log Z-(log Z)<<N where the distribution is exponential.