Muneki Yasuda

Approximate learning algorithm in boltzmann machines

Boltzmann machines can be regarded as Markov random fields. For binary cases, they are equivalent to the Ising spin model in statistical mechanics. Learning systems in Boltzmann machines are one of the NP-hard problems. Thus, in general we have to use approximate methods to construct practical learning algorithms in this context. In this letter, we propose new and practical learning algorithms for Boltzmann machines by using the belief propagation algorithm and the linear response approximation, which are often referred as advanced mean field methods. Finally, we show the validity of our algorithm using numerical experiments.

Susceptibility propagation by using diagonal consistency.

A susceptibility propagation that is constructed by combining a belief propagation and a linear response method is used for approximate computation for Markov random fields. Herein, we formulate an improved susceptibility propagation by using the concept of a diagonal matching method that is based on mean-field approaches to inverse Ising problems. The proposed susceptibility propagation is robust for various network structures, and it is reduced to the ordinary susceptibility propagation and to the adaptive Thouless-Anderson-Palmer equation in special cases.

Replica Analysis of Preferential Urn Model

We analyze a preferential urn model with randomness using the replica method. The preferential urn model is a stochastic model based on the concept “the rich get richer.” In the replica analysis, we clarify that the preferential urn model with randomness shows a fat-tailed occupation distribution. We discuss the intuitive picture of the preferential urn model, a relationship between quenched and annealed systems, and a trial for the application of the preferential urn model to econophysics. The analytical treatments and results would be useful for various research fields such as complex networks, stochastic models, and econophysics.

Inverse Problem in Pairwise Markov Random Fields Using Loopy Belief Propagation

In this paper, using Chebyshev polynomials, we derive a new representation of loopy belief propagation for pairwise Markov random fields in terms of moments. Using the new representation, we propose a new method to solve a statistical inverse problem in pairwise Markov random fields within the framework of loopy belief propagation. Our method allows us to solve the statistical inverse problem without any iterative operations. We numerically verify our method for the statistical inverse problem in Q -state Potts models.

Statistical performance analysis by loopy belief propagation in Bayesian image modeling

The mathematical structures of loopy belief propagation are reviewed for Bayesian image modeling from the standpoint of statistical mechanical informatics. We propose some schemes for evaluating the statistical performance of probabilistic binary image restoration. The schemes are constructed by means of the LBP, which is known as the Bethe approximation in statistical mechanics. We show some results of numerical experiments obtained by using the LBP algorithm as well as the statistical performance analysis for the probabilistic image restorations.

Statistical analysis of loopy belief propagation in random fields.

Loopy belief propagation (LBP), which is equivalent to the Bethe approximation in statistical mechanics, is a message-passing-type inference method that is widely used to analyze systems based on Markov random fields (MRFs). In this paper, we propose a message-passing-type method to analytically evaluate the quenched average of LBP in random fields by using the replica cluster variation method. The proposed analytical method is applicable to general pairwise MRFs with random fields whose distributions differ from each other and can give the quenched averages of the Bethe free energies over random fields, which are consistent with numerical results. The order of its computational cost is equivalent to that of standard LBP. In the latter part of this paper, we describe the application of the proposed method to Bayesian image restoration, in which we observed that our theoretical results are in good agreement with the numerical results for natural images.

A generalization of improved susceptibility propagation

Recently, an effective susceptibility propagation method for binary Markov random fields based on a concept of diagonal consistency was proposed. This improved susceptibility propagation is a powerful method and exhibits a robust performance for various types of network structures. In this paper, a generalization of the improved susceptibility propagation using an orthonormal function expansion is proposed in which any pairwise potential functions and multivalued random variables are acceptable. In the latter part of this paper, the proposed method is applied to a direct problem and an inverse problem on a generalized sparse prior, which is a recently proposed prior model for natural images.

Statistical analysis of the expectation-maximization algorithm with loopy belief propagation in Bayesian image modeling

New statistical approaches to hyperparameter estimation by means of the expectation-maximization (EM) algorithm and loopy belief propagation (LBP) are given for Bayesian image modeling from the standpoint of statistical-mechanical informatics. In the present paper, we give a new scheme for computing the average of the trajectory in the EM algorithm with LBP with respect to all the possible degraded images generated by the assumed degradation process from a given original image. Moreover, we also give a new scheme for computing the average of the trajectory in the EM algorithm with LBP with respect to all the possible original images generated by the assumed prior probability distribution. Bayesian image models for binary image restorations are constructed as Ising models with random external fields and spatially uniform nearest-neighbor interactions on the square lattice. Our schemes for computing the above random averages for some statistical quantities in LBP are constructed by means of the Bethe approximation for random spin systems. They are reduced to the solution of simultaneous integral equations for distributions of messages in LBP. We show the results of some numerical experiments in LBP as well as the statistical analysis of the trajectory in the EM algorithm for binary image restoration by LBP.

TAP equation for non-negative Boltzmann machine

Mean-field methods for spin systems are frequently used in not only statistical physics but also information sciences. We focus on the Plefka expansion method for spin systems with two-body interactions. The Plefka expansion is a useful perturbative expansion of the Gibbs free energy, and it can systematically provide the naive mean-field approximation, the Thouless–Anderson–Palmer (TAP) equation and higher-order approximations. In the first part of this paper, using the linear response relation, we derive a recurrence formula for perturbative coefficients in the Plefka expansion. Our recurrence formula enables us to systematically derive general order coefficients. In the latter part of the paper, we apply our recurrence formula to the non-negative Boltzmann machine in which all spin variables are constrained to have non-negative real values, and we obtain the TAP equation for this model. We verify the performance of our TAP equation by conducting some numerical experiments.

Statistical Analysis of Gaussian Image Inpainting Problems

Bayesian image processing is one of the activity research fields in computer science. Some of the approaches can often be reduced to inference problems on Markov random fields (MRFs). Image inpainting is one such problem. Image inpainting is used to infer intensities in missing regions from an observed image in which the intensities of some pixels are missing. In the inference scheme of inpainting, each observed image is divided into missing regions and nonmissing regions, and it is assumed that it is known which region each pixel belongs to in advance. A deterministic algorithm for the image inpainting problem based on Gaussian MRFs has been proposed. The proposed algorithm returns a unique output image for each observed image. By considering variations among observed images, we can define the statistical performances of the deterministic algorithm. The calculation of statistical performances corresponds to taking the average over all the possible observed images. Statistical performance analysis in probabilistic inference systems is one of the topics of statistical–mechanical informatics, and there are many successful results regarding statistical performance analysis in various research fields of computer sciences. Nishimori and Wong calculated the statistical performance in binary image restorations using MRFs defined on a complete graph. Statistical performance analysis in image restorations using Gaussian MRFs has also been calculated accurately by using the multi-dimensional Gauss integral formulas. Kataoka et al. have proposed approximate methods for evaluating the statistical performances and applied them for binary image restorations using MRFs on square grid graphs. In the present paper, we evaluate statistical performances of image inpaintings by Gaussian MRFs. Our evaluation method is an extension of the idea to evaluate the approximate statistical performance based on the loopy belief propagation in ref. 8, and the calculation of statistical performance is reduced to the problem of solving simultaneous integral equations of distributions of estimated values. We consider an image composed of N pixels on a square grid graph and define a state, corresponding to light intensities, for each pixel as any real number in ð 1;þ1Þ. N pixels are divided into missing regions and non-missing regions. Suppose that x 1⁄4 fxig and y 1⁄4 fyjg are sets of pixels in missing regions and non-missing regions, respectively. Here, i is a label at each pixel belonging to missing regions, whereas j is associated to a pixel in non-missing regions. In this paper, we consider an image in which each pixel is randomly missing with probability p. Let c 1⁄4 fcig; ci 2 f0; 1g; i 1⁄4 1; . . . ; N be a set of state variables that indicate whether pixels are missing or not. ci 1⁄4 0 denotes a pixel i belonging to non-missing regions and ci 1⁄4 1 denotes a pixel i belonging to missing regions. We assume that the posterior probability density function of x, under both c and y are given, is denoted by Pðx j c; yÞ 1⁄4 exp1⁄2 Hðx j c; yÞ =Z, where the energy function Hðx j c; yÞ is defined as Hðx j c; yÞ 1⁄4 1 2 X ði;jÞ2E 2 cicjðxi xjÞ