Kazuho Watanabe

Deterministic annealing variant of variational Bayes method

The Variational Bayes (VB) method is widely used as an approximation of the Bayesian method. Because the VB method is a gradient algorithm, it is often trapped by poor local optimal solutions. We introduce deterministic annealing to the VB method to overcome such a local optimal problem. A temperature parameter is introduced to the free energy for controlling the annealing process deterministically. Applying the method to a mixture of Gaussian models and hidden Markov models, we show that it can obtain the global optimum of the free energy and discover optimal model structure.

Stochastic complexity of variational Bayesian hidden Markov models

Variational Bayesian learning was proposed as the approximation method of Bayesian learning. Inspite of efficiency and experimental good performance, their mathematical property has not yet been clarified. In this paper we analyze variational Bayesian hidden Markov models which include the true one thus the models are non-identifiable. We derive their asymptotic stochastic complexity. It is shown that, in some prior condition, the stochastic complexity is much smaller than those of identifiable models.

Stochastic complexities of general mixture models in variational Bayesian learning

In this paper, we focus on variational Bayesian learning of general mixture models. Variational Bayesian learning was proposed as an approximation of Bayesian learning. While it has provided computational tractability and good generalization in many applications, little has been done to investigate its theoretical properties. The asymptotic form was obtained for the stochastic complexity, or the free energy in the variational Bayesian learning of a mixture of exponential-family distributions, which is the main contribution this paper makes. We reveal that the stochastic complexities become smaller than those of regular statistical models, which implies that the advantages of Bayesian learning are still retained in variational Bayesian learning. Moreover, the derived bounds indicate what influence the hyperparameters have on the learning process, and the accuracy of the variational Bayesian approach as an approximation of true Bayesian learning.

Upper bound for variational free energy of Bayesian networks

In recent years, variational Bayesian learning has been used as an approximation of Bayesian learning. In spite of the computational tractability and good generalization in many applications, its statistical properties have yet to be clarified. In this paper, we focus on variational Bayesian learning of Bayesian networks which are widely used in information processing and uncertain artificial intelligence. We derive upper bounds for asymptotic variational free energy or stochastic complexities of bipartite Bayesian networks with discrete hidden variables. Our result theoretically supports the effectiveness of variational Bayesian learning as an approximation of Bayesian learning.

Variational Bayesian Mixture Model on a Subspace of Exponential Family Distributions

Exponential principal component analysis (e-PCA) has been proposed to reduce the dimension of the parameters of probability distributions using Kullback information as a distance between two distributions. It also provides a framework for dealing with various data types such as binary and integer for which the Gaussian assumption on the data distribution is inappropriate. In this paper, we introduce a latent variable model for the e-PCA. Assuming the discrete distribution on the latent variable leads to mixture models with constraint on their parameters. This provides a framework for clustering on the lower dimensional subspace of exponential family distributions. We derive a learning algorithm for those mixture models based on the variational Bayes (VB) method. Although intractable integration is required to implement the algorithm for a subspace, an approximation technique using Laplaces method allows us to carry out clustering on an arbitrary subspace. Combined with the estimation of the subspace, the resulting algorithm performs simultaneous dimensionality reduction and clustering. Numerical experiments on synthetic and real data demonstrate its effectiveness for extracting the structures of data as a visualization technique and its high generalization ability as a density estimation model.

Bayesian properties of normalized maximum likelihood and its fast computation

The normalized maximized likelihood (NML) provides the minimax regret solution in universal data compression, gambling, and prediction, and it plays an essential role in the minimum description length (MDL) method of statistical modeling and estimation. Here we show that when the sample space is finite, a generic condition on the linear independence of the component models implies that the normalized maximum likelihood has an exact Bayes-like representation as a mixture of the component models, even in finite samples, though the weights of linear combination may be both positive and negative. This addresses in part the relationship between MDL and Bayes modeling. The representation also has the practical advantage of speeding the calculation of marginals and conditionals required for coding and prediction applications.

Stochastic complexity for mixture of exponential families in variational bayes

The Variational Bayesian learning, proposed as an approximation of the Bayesian learning, has provided computational tractability and good generalization performance in many applications. However, little has been done to investigate its theoretical properties. In this paper, we discuss the Variational Bayesian learning of the mixture of exponential families and derive the asymptotic form of the stochastic complexities. We show that the stochastic complexities become smaller than those of regular statistical models, which implies the advantage of the Bayesian learning still remains in the Variational Bayesian learning. Stochastic complexity, which is called the marginal likelihood or the free energy, not only becomes important in addressing the model selection problem but also enables us to discuss the accuracy of the Variational Bayesian approach as an approximation of the true Bayesian learning.

Spectral-Based Contractible Parallel Coordinates

Parallel coordinates is well-known as a popular tool for visualizing the underlying relationships among variables in high-dimension datasets. However, this representation still suffers from visual clutter arising from intersections among poly line plots especially when the number of data samples and their associated dimension become high. This paper presents a method of alleviating such visual clutter by contracting multiple axes through the analysis of correlation between every pair of variables. In this method, we first construct a graph by connecting axis nodes with an edge weighted by data correlation between the corresponding pair of dimensions, and then reorder the multiple axes by projecting the nodes onto the primary axis obtained through the spectral graph analysis. This allows us to compose a dendrogram tree by recursively merging a pair of the closest axes one by one. Our visualization platform helps the visual interpretation of such axis contraction by plotting the principal component of each data sample along the composite axis. Smooth animation of the associated axis contraction and expansion has also been implemented to enhance the visual readability of behavior inherent in the given high-dimensional datasets.

Divergence measures and a general framework for local variational approximation

The local variational method is a technique to approximate an intractable posterior distribution in Bayesian learning. This article formulates a general framework for local variational approximation and shows that its objective function is decomposable into the sum of the Kullback information and the expected Bregman divergence from the approximating posterior distribution to the Bayesian posterior distribution. Based on a geometrical argument in the space of approximating posteriors, we propose an efficient method to evaluate an upper bound of the marginal likelihood. Moreover, we demonstrate that the variational Bayesian approach for the latent variable models can be viewed as a special case of this general framework.

Variational Bayesian Inference Algorithms for Infinite Relational Model of Network Data

Network data show the relationship among one kind of objects, such as social networks and hyperlinks on the Web. Many statistical models have been proposed for analyzing these data. For modeling cluster structures of networks, the infinite relational model (IRM) was proposed as a Bayesian nonparametric extension of the stochastic block model. In this brief, we derive the inference algorithms for the IRM of network data based on the variational Bayesian (VB) inference methods. After showing the standard VB inference, we derive the collapsed VB (CVB) inference and its variant called the zeroth-order CVB inference. We compared the performances of the inference algorithms using six real network datasets. The CVB inference outperformed the VB inference in most of the datasets, and the differences were especially larger in dense networks.