Haiping Huang

Origin of the computational hardness for learning with binary synapses

Through supervised learning in a binary perceptron one is able to classify an extensive number of random patterns by a proper assignment of binary synaptic weights. However, to find such assignments in practice is quite a nontrivial task. The relation between the weight space structure and the algorithmic hardness has not yet been fully understood. To this end, we analytically derive the Franz-Parisi potential for the binary perceptron problem by starting from an equilibrium solution of weights and exploring the weight space structure around it. Our result reveals the geometrical organization of the weight space; the weight space is composed of isolated solutions, rather than clusters of exponentially many close-by solutions. The pointlike clusters far apart from each other in the weight space explain the previously observed glassy behavior of stochastic local search heuristics.

The Network Source Location Problem: Ground State Energy, Entropy and Effects of Freezing

Ground state entropy of the network source location problem is evaluated at both the replica symmetric level and one-step replica symmetry breaking level using the entropic cavity method. The regime that is a focus of this study, is closely related to the vertex cover problem with randomly quenched covered nodes. The resulting entropic message passing inspired decimation and reinforcement algorithms are used to identify the optimal location of sources in single instances of transportation networks. The conventional belief propagation without taking the entropic effect into account is also compared. We find that in the glassy phase the entropic message passing inspired decimation yields a lower ground state energy compared to the belief propagation without taking the entropic effect. Using the extremal optimization algorithm, we study the ground state energy and the fraction of frozen hubs, and extend the algorithm to collect statistics of the entropy. The theoretical results are compared with the extremal optimization results.

Adaptive Thouless-Anderson-Palmer approach to inverse Ising problems with quenched random fields.

The adaptive Thouless-Anderson-Palmer equation is derived for inverse Ising problems in the presence of quenched random fields. We test the proposed scheme on Sherrington-Kirkpatrick, Hopfield, and random orthogonal models and find that the adaptive Thouless-Anderson-Palmer approach allows accurate inference of quenched random fields whose distribution can be either Gaussian or bimodal. In particular, another competitive method for inferring external fields, namely, the naive mean field method with diagonal weights, is compared and discussed.

Entropy landscape of solutions in the binary perceptron problem

The statistical picture of the solution space for a binary perceptron is studied. The binary perceptron learns a random classification of input random patterns by a set of binary synaptic weights. The learning of this network is difficult especially when the pattern (constraint) density is close to the capacity, which is supposed to be intimately related to the structure of the solution space. The geometrical organization is elucidated by the entropy landscape from a reference configuration and of solution-pairs separated by a given Hamming distance in the solution space. We evaluate the entropy at the annealed level as well as replica symmetric level and the mean field result is confirmed by the numerical simulations on single instances using the proposed message passing algorithms. From the first landscape (a random configuration as a reference), we see clearly how the solution space shrinks as more constraints are added. From the second landscape of solution-pairs, we deduce the coexistence of clustering and freezing in the solution space.

Combined local search strategy for learning in networks of binary synapses

Learning in networks of binary synapses is known to be an NP-complete problem. A combined stochastic local search strategy in the synaptic weight space is constructed to further improve the learning performance of a single random walker. We apply two correlated random walkers guided by their Hamming distance and associated energy costs (the number of unlearned patterns) to learn a same large set of patterns. Each walker first learns a small part of the whole pattern set (partially different for both walkers but with the same amount of patterns) and then both walkers explore their respective weight spaces cooperatively to find a solution to classify the whole pattern set correctly. The desired solutions locate at the common parts of weight spaces explored by these two walkers. The efficiency of this combined strategy is supported by our extensive numerical simulations and the typical Hamming distance as well as energy cost is estimated by an annealed computation. Copyright (C) EPLA, 2011

Message passing algorithms for the Hopfield network reconstruction: Threshold behavior and limitation

The Hopfield network is reconstructed as an inverse Ising problem by passing messages. The applied susceptibility propagation algorithm is shown to improve significantly on other mean-field-type methods and extends well into the low-temperature region. However, this iterative algorithm is limited by the nature of the supplied data. Its performance deteriorates as the data become highly magnetized, and this method finally fails in the presence of the frozen type data where at least two of its magnetizations are equal to 1 in absolute value. On the other hand, a threshold behavior is observed for the susceptibility propagation algorithm and the transition from good reconstruction to poor one becomes sharper as the network size increases.

Learning by random walks in the weight space of the Ising perceptron

Several variants of a stochastic local search process for constructing the synaptic weights of an Ising perceptron are studied. In this process, binary patterns are sequentially presented to the Ising perceptron and are then learned as the synaptic weight configuration is modified through a chain of single- or double-weight flips within the compatible weight configuration space of the earlier learned patterns. This process is able to reach a storage capacity of alpha approximate to 0.63 for pattern length N = 101 and alpha approximate to 0.41 for N = 1001. If in addition a relearning process is exploited, the learning performance is further improved to a storage capacity of alpha approximate to 0.80 for N = 101 and alpha approximate to 0.42 for N = 1001. We found that, for a given learning task, the solutions constructed by the random walk learning process are separated by a typical Hamming distance, which decreases with the constraint density a of the learning task; at a fixed value of a, the width of the Hamming distance distribution decreases with N.

Cavity approach to the Sourlas code system

The statistical physics properties of regular and irregular Sourlas codes are investigated in this paper by the cavity method. At finite temperatures, the free-energy density of these coding systems is derived and compared with the result obtained by the replica method. In the zero-temperature limit, the Shannons bound is recovered in the case of infinite-body interactions while the code rate is still finite. However, the decoding performance as obtained by the replica theory has not considered the zero-temperature entropic effect. The cavity approach is able to consider the ground-state entropy. It leads to a set of evanescent cavity fields propagation equations which further improve the decoding performance as confirmed by our numerical simulations on single instances. For the irregular Sourlas code, we find that it takes the trade-off between good dynamical property and high performance of decoding. In agreement with the results found from the algorithmic point of view, the decoding exhibits a first-order phase transition as occurs in the regular code system with three-body interactions. The cavity approach for the Sourlas code system can be extended to consider first-step replica symmetry breaking.

State sampling dependence of hopfield network inference

The fully connected Hopfield network is inferred based on observed magnetizations and pairwise correlations. We present the system in the glassy phase with low temperature and high memory load. We find that the inference error is very sensitive to the form of state sampling. When a single state is sampled to compute magnetizations and correlations, the inference error is almost indistinguishable irrespective of the sampled state. However, the error can be greatly reduced if the data is collected with state transitions. Our result holds for different disorder samples and accounts for the previously observed large fluctuations of inference error at low temperatures.

Counting solutions from finite samplings.

We formulate the solution counting problem within the framework of the inverse Ising problem and use fast belief propagation equations to estimate the entropy whose value provides an estimate of the true one. We test this idea on both diluted models [random 2-SAT (2-satisfiability) and 3-SAT problems] and a fully connected model (binary perceptron), and show that when the constraint density is small, this estimate can be very close to the true value. The information stored by the salamander retina under the natural movie stimuli can also be estimated, and our result is consistent with that obtained by the Monte Carlo method. Of particular significance is that the sizes of other metastable states for this real neuronal network are predicted.