Chulan Kwon

Storage capacity of a fully-connected parity machine with continuous weights

We study a fully-connected parity machine with K hidden units for continuous weights. The geometrical structure of the weight space of this model is analysed in terms of the volumes associated with the internal representations of the training set. By examining the asymptotic behaviour of order parameters in the large K limit, we find the maximum number , the storage capacity, of patterns per input unit to be up to leading order, which saturates the mathematical bound given by Mitchison and Durbin. Unlike the committee machine, the storage capacity per weight remains unchanged compared with the corresponding tree-like architecture.

One-step replica-symmetry-breaking solution for a perceptron learning with weight mismatch

We investigate the properties of the one-step replica-symmetry-breaking (1RSB) solution for a perceptron learning from examples with weight mismatch where the entropy zero line crosses the Almeida - Thouless (AT) line of the RS solution. For a small number of examples we find the optimal 1RSB solution which has the maximum free energy, non-negative entropy and satisfies the stability condition, the AT criterion for the 1RSB solution. The transition from RS to 1RSB is continuous or discontinuous depending on whether the RS AT line is above or below the zero entropy line. However, for a relatively large number of examples, the 1RSB solution which maximizes the free energy becomes unstable, and should be replaced by higher-step RSB solutions. We also obtain the AT line for the 1RSB solution.

Generalization in a perceptron with a sigmoid transfer function

Learning of layered neural networks is studied using the methods of statistical mechanics. Networks are trained from examples using the Gibbs algorithm. We focus on the generalization curve, i.e. the average generalization error as a function of the number of the examples. We consider perceptron learning with a sigmoid transfer function. Ising perceptrons, with weights constrained to be discrete, exhibit sudden learning at low temperatures within the annealed approximation. There is a first order transition from a state of poor generalization to a state of perfect generalization. When the transfer function is smooth, the first order transition occurs only at low temperatures. The transition becomes continuous at high temperatures. When the transfer function is steep, the first order transition line is extended to the higher temperature. The analytic results show a good agreement with the computer simulations.