Ronny Meir

Storing and Retrieving Information in a Layered Spin System

We introduce a neural network model with layered architecture and binary (spin) variables. Hebbian rules are used to define unidirectional couplings between spins of adjacent layers. A fast learning algorithm produces couplings that store a large number of random patterns, and efficiently recognizes noisy patterns. Performance of this network is compared with spin-glass type models of pattern recognition.

Layered neural networks

Some of the recent work done on layered feed-forward networks is reviewed. First we describe exact solutions for the dynamics of such networks, which are expected to respond to an input by going through a sequence of preassigned states on the various layers. The family of networks considered has a variety of interlayer couplings: linear and nonlinear Hebbian, Hebbian with Gaussian synaptic noise and with various kinds of dilution, and the pseudoinverse (projector) matrix of couplings. In all cases our solutions take the form of layer-to-layer recursions for the mean overlap with a (random) key pattern and for the width of the embedding field distribution. Dynamics is governed by the fixed points of these recursions. For all cases nontrivial domains of attraction of the memory states are found. Next we review studies of unsupervised leaming in such networks and the emergence of orientation-selective cells. Finally the main ideas of three supervised leaming procedures, recendy introduced for layered networks, are oudined. All three procedures are based on a search in the space of intemal representations; one is designed for leaming in networks with fixed architecture and has no associated convergence theorem, whereas the other two are guaranteed to converge but may require expansion of the network by an uncontrolled number of hidden units.