Ayaka Sakata

Phase Transitions and Sample Complexity in Bayes-Optimal Matrix Factorization

We analyze the matrix factorization problem. Given a noisy measurement of a product of two matrices, the problem is to estimate back the original matrices. It arises in many applications, such as dictionary learning, blind matrix calibration, sparse principal component analysis, blind source separation, low rank matrix completion, robust principal component analysis, or factor analysis. It is also important in machine learning: unsupervised representation learning can often be studied through matrix factorization. We use the tools of statistical mechanics-the cavity and replica methods-to analyze the achievability and computational tractability of the inference problems in the setting of Bayes-optimal inference, which amounts to assuming that the two matrices have random-independent elements generated from some known distribution, and this information is available to the inference algorithm. In this setting, we compute the minimal mean-squared-error achievable, in principle, in any computational time, and the error that can be achieved by an efficient approximate message passing algorithm. The computation is based on the asymptotic state-evolution analysis of the algorithm. The performance that our analysis predicts, both in terms of the achieved mean-squared-error and in terms of sample complexity, is extremely promising and motivating for a further development of the algorithm.

Funnel Landscape and Mutational Robustness as a Result of Evolution under Thermal Noise

Using a statistical-mechanical model of spins, the evolution of phenotype dynamics is studied. Configurations of spins and their interaction J represent the phenotype and genotype, respectively. The fitness for selection of J is given by the equilibrium spin configurations determined by a Hamiltonian with J under thermal noise. The genotype J evolves through mutational changes under selection pressure to raise its fitness value. From Monte Carlo simulations we find that the frustration around the target spins disappears for J evolved under temperature beyond a certain threshold. The evolved Js give the funnel-like dynamics, which is robust to noise and also to mutation.

Statistical mechanics of dictionary learning

Finding a basis matrix (dictionary) by which objective signals are represented sparsely is of major relevance in various scientific and technological fields. We consider a problem to learn a dictionary from a set of training signals. We employ techniques of statistical mechanics of disordered systems to evaluate the size of the training set necessary to typically succeed in the dictionary learning. The results indicate that the necessary size is much smaller than previously estimated, which theoretically supports and/or encourages the use of dictionary learning in practical situations.

Sample complexity of Bayesian optimal dictionary learning

We consider a learning problem of identifying a dictionary matrix D ∈ R^M×N from a sample set of M dimensional vectors Y ∈ R^M×P = N^-1/2DX ∈ R^M×P, where X ∈ R^N×p is a sparse matrix in which the density of non-zero entries is 0 <; ρ <; 1. In particular, we focus on the minimum sample size P_c (sample complexity) necessary for perfectly identifying D of the optimal learning scheme when D and X are independently generated from certain distributions. By using the replica method of statistical mechanics, we show that P_c ~ O(N) holds as long as α = M/N > ρ is satisfied in the limit of N → ∞. Our analysis also implies that the posterior distribution given Y is condensed only at the correct dictionary D when the compression rate α is greater than a certain critical value α_M(p). This suggests that belief propagation may allow us to learn D with a low computational complexity using O(N) samples.

Replica symmetry breaking in an adiabatic spin-glass model of adaptive evolution

We study evolutionary canalization using a spin-glass model with replica theory, where spins and their interactions are dynamic variables whose configurations correspond to phenotypes and genotypes, respectively. The spins are updated under temperature TS, and the genotypes evolve under temperature TJ, according to the evolutionary fitness. It is found that adaptation occurs at TS < TRSS, and a replica symmetric phase emerges at TRSBS < TS < TRSS. The replica symmetric phase implies canalization, and replica symmetry breaking at lower temperatures indicates loss of robustness.

Approximate message passing for nonconvex sparse regularization with stability and asymptotic analysis

We analyse a linear regression problem with nonconvex regularization called smoothly clipped absolute deviation (SCAD) under an overcomplete Gaussian basis for Gaussian random data. We propose an approximate message passing (AMP) algorithm considering nonconvex regularization, namely SCAD-AMP, and analytically show that the stability condition corresponds to the de Almeida--Thouless condition in spin glass literature. Through asymptotic analysis, we show the correspondence between the density evolution of SCAD-AMP and the replica symmetric solution. Numerical experiments confirm that for a sufficiently large system size, SCAD-AMP achieves the optimal performance predicted by the replica method. Through replica analysis, a phase transition between replica symmetric (RS) and replica symmetry breaking (RSB) region is found in the parameter space of SCAD. The appearance of the RS region for a nonconvex penalty is a significant advantage that indicates the region of smooth landscape of the optimization problem. Furthermore, we analytically show that the statistical representation performance of the SCAD penalty is better than that of L1-based methods, and the minimum representation error under RS assumption is obtained at the edge of the RS/RSB phase. The correspondence between the convergence of the existing coordinate descent algorithm and RS/RSB transition is also indicated.

Time evolution of the autocorrelation function in dynamical replica theory

Asynchronous dynamics given by the master equation in the Sherrington–Kirkpatrick (SK) spin-glass model is studied based on dynamical replica theory (DRT) with an extension to take into account the autocorrelation function. The dynamical behaviour of the system is approximately described by dynamical equations of the macroscopic quantities: magnetization, energy contributed by randomness and the autocorrelation function. The dynamical equations under the replica symmetry assumption are derived by introducing the subshell equipartitioning assumption and exploiting the replica method. The obtained dynamical equations are compared with Monte Carlo simulations, and it is demonstrated that the proposed formula describes well the time evolution of the autocorrelation function in some parameter regions. The study offers a reasonable description of the autocorrelation function in the SK spin-glass system.

Partial annealing of a coupled mean-field spin-glass model with an embedded pattern.

A partially annealed mean-field spin-glass model with a locally embedded pattern is studied. The model consists of two dynamical variables, spins and interactions, that are in contact with thermal baths at temperatures T(S) and T(J), respectively. Unlike the quenched system, characteristic correlations among the interactions are induced by the partial annealing. The model exhibits three phases: paramagnetic, ferromagnetic and spin-glass. In the ferromagnetic phase, the embedded pattern is stably realized. The phase diagram depends significantly on the ratio of the two temperatures, n=T(S)/T(J). In particular, a reentrant transition from the embedded ferromagnetic to the spin-glass phase with T(S) decreasing is found only below a certain value of n. This indicates that above the critical value n(c) the embedded pattern is supported by a local field from a nonembedded region. Some equilibrium properties of the interactions in the partial annealing are also discussed in terms of frustration.

Statistical-mechanical study of evolution of robustness in noisy environments

In biological systems, expression dynamics that can provide fitted phenotype patterns with respect to a specific function have evolved through mutations. This has been observed in the evolution of proteins for realizing folding dynamics through which a target structure is shaped. We study this evolutionary process by introducing a statistical-mechanical model of interacting spins, where a configuration of spins and their interactions J represent a phenotype and genotype, respectively. The phenotype dynamics are given by a stochastic process with temperature TS under a Hamiltonian with J. The evolution of J is also stochastic with temperature TJ and follows mutations introduced into J and selection based on a fitness defined for a configuration of a given set of target spins. Below a certain temperature TS(c2), the interactions J that achieve the target pattern evolve, whereas another phase transition is observed at TS(c1)<TS(c2). At low temperatures TS<TS(c1), the Hamiltonian exhibits a spin-glass-like phase, where the dynamics toward the target pattern require long time steps, and the fitness often decreases drastically as a result of a single mutation to J. In the intermediate-temperature region, the dynamics to shape the target pattern proceed rapidly and are robust to mutations of J. The interactions in this region have no frustration around the target pattern and results in funnel-type dynamics. We propose that the ubiquity of funnel-type dynamics, as observed in protein folding, is a consequence of evolution subjected to thermal noise beyond a certain level; this also leads to mutational robustness of the fitness.

Estimator of prediction error based on approximate message passing for penalized linear regression

We propose an estimator of prediction error using an approximate message passing (AMP) algorithm that can be applied to a broad range of sparse penalties. Following Steins lemma, the estimator of the generalized degrees of freedom, which is a key quantity for the construction of the estimator of the prediction error, is calculated at the AMP fixed point. The resulting form of the AMP-based estimator does not depend on the penalty function, and its value can be further improved by considering the correlation between predictors. The proposed estimator is asymptotically unbiased when the components of the predictors and response variables are independently generated according to a Gaussian distribution. We examine the behaviour of the estimator for real data under nonconvex sparse penalties, where Akaikes information criterion does not correspond to an unbiased estimator of the prediction error. The model selected by the proposed estimator is close to that which minimizes the true prediction error.