Naonori Ueda

SMEM Algorithm for Mixture Models

We present a split-and-merge expectation-maximization (SMEM) algorithm to overcome the local maxima problem in parameter estimation of finite mixture models. In the case of mixture models, local maxima often involve having too many components of a mixture model in one part of the space and too few in another, widely separated part of the space. To escape from such configurations, we repeatedly perform simultaneous split-and-merge operations using a new criterion for efficiently selecting the split-and-merge candidates. We apply the proposed algorithm to the training of gaussian mixtures and mixtures of factor analyzers using synthetic and real data and show the effectiveness of using the split- and-merge operations to improve the likelihood of both the training data and of held-out test data. We also show the practical usefulness of the proposed algorithm by applying it to image compression and pattern recognition problems.

Optimal linear combination of neural networks for improving classification performance

This paper presents a new method for linearly combining multiple neural network classifiers based on the statistical pattern recognition theory. In our approach, several neural networks are first selected based on which works best for each class in terms of minimizing classification errors. Then, they are linearly combined to form an ideal classifier that exploits the strengths of the individual classifiers. In this approach, the minimum classification error criterion is utilized to estimate the optimal linear weights. In this formulation, because the classification decision rule is incorporated into the cost function, a more suitable better combination of weights for the classification objective could be obtained. Experimental results using artificial and real data sets show that the proposed method can construct a better combined classifier that outperforms the best single classifier in terms of overall classification errors for test data.

Generalization error of ensemble estimators

It has been empirically shown that a better estimate with less generalization error can be obtained by averaging outputs of multiple estimators. This paper presents an analytical result for the generalization error of ensemble estimators. First, we derive a general expression of the ensemble generalization error by using factors of interest (bias, variance, covariance, and noise variance) and show how the generalization error is affected by each of them. Some special cases are then investigated. The result of a simulation is shown to verify our analytical result. A practically important problem of the ensemble approach, ensemble dilemma, is also discussed.

Bayesian model search for mixture models based on optimizing variational bounds

When learning a mixture model, we suffer from the local optima and model structure determination problems. In this paper, we present a method for simultaneously solving these problems based on the variational Bayesian (VB) framework. First, in the VB framework, we derive an objective function that can simultaneously optimize both model parameter distributions and model structure. Next, focusing on mixture models, we present a deterministic algorithm to approximately optimize the objective function by using the idea of the split and merge operations which we previously proposed within the maximum likelihood framework. Then, we apply the method to mixture of expers (MoE) models to experimentally show that the proposed method can find the optimal number of experts of a MoE while avoiding local maxima.

Variational bayesian estimation and clustering for speech recognition

In this paper, we propose variational Bayesian estimation and clustering for speech recognition (VBEC), which is based on the variational Bayesian (VB) approach. VBEC is a total Bayesian framework: all speech recognition procedures (acoustic modeling and speech classification) are based on VB posterior distribution, unlike the maximum likelihood (ML) approach based on ML parameters. The total Bayesian framework generates two major Bayesian advantages over the ML approach for the mitigation of over-training effects, as it can select an appropriate model structure without any data set size condition, and can classify categories robustly using a predictive posterior distribution. By using these advantages, VBEC: 1) allows the automatic construction of acoustic models along two separate dimensions, namely, clustering triphone hidden Markov model states and determining the number of Gaussians and 2) enables robust speech classification, based on Bayesian predictive classification using VB posterior distributions. The capabilities of the VBEC functions were confirmed in large vocabulary continuous speech recognition experiments for read and spontaneous speech tasks. The experiments confirmed that VBEC automatically constructed accurate acoustic models and robustly classified speech, i.e., totally mitigated the over-training effects with high word accuracies due to the VBEC functions.

A new competitive learning approach based on an equidistortion principle for designing optimal vector quantizers

A new competitive learning approach is presented for optimal vector quantizer design. First, it is shown that the original CL algorithm is equivalent to the traditional nonconnectionist VQ design algorithm called the LBG algorithm. Then, it is shown that the conventional conscience principle or equiprobable principle is not optimal from the standpoint of the minimization of the expected distortion. Next, a basic principle called the equidistortion principle for the design of optimal vector quantizers is theoretically derived by using Gershos asymptotic theory. This paper proposes a new competitive learning algorithm with a selection mechanism, called the CSL (competitive and selective learning) algorithm, which is based on the equidistortion principle. Because the selection mechanism enables the system to escape from local minima, the proposed algorithm can obtain better performance without a particular initialization procedure even when the input data cluster in a number of regions in the input vector space. Simulation results comparing the performance of the CSL algorithm with other conventional algorithms for synthetic and real-world data show that the CSL algorithm, in spite of its simplicity, always produces the best quantizers with the least distortion, regardless of the initial codes. The optimality of the proposed CSL algorithm is also verified through a synthetic one-dimensional quantizer problem.

Learning visual models from shape contours using multiscale convex/concave structure matching

A novel approach is proposed for learning a visual model from real shape samples of the same class. The approach can directly acquire a visual model by generalizing the multiscale convex/concave structure of a class of shapes, that is, the approach is based on the concept that shape generalization is shape simplification wherein perceptually relevant features are retained. The simplification does not mean the approximation of shapes but rather the extraction of the optimum scale convex/concave structure common to shape samples of the class. The common structure is obtained by applying the multiscale convex/concave structure-matching method to all shape pairs among given shape samples of the class and by integrating the matching results. The matching method, is applicable to heavily deformed shapes and is effectively implemented with dynamic programming techniques. The approach can acquire a visual model from a few samples without any a priori knowledge of the class. The obtained model is very useful for shape recognition. Results of applying the proposed method are presented. >

Multichannel Extensions of Non-Negative Matrix Factorization With Complex-Valued Data

This paper presents new formulations and algorithms for multichannel extensions of non-negative matrix factorization (NMF). The formulations employ Hermitian positive semidefinite matrices to represent a multichannel version of non-negative elements. Multichannel Euclidean distance and multichannel Itakura-Saito (IS) divergence are defined based on appropriate statistical models utilizing multivariate complex Gaussian distributions. To minimize this distance/divergence, efficient optimization algorithms in the form of multiplicative updates are derived by using properly designed auxiliary functions. Two methods are proposed for clustering NMF bases according to the estimated spatial property. Convolutive blind source separation (BSS) is performed by the multichannel extensions of NMF with the clustering mechanism. Experimental results show that 1) the derived multiplicative update rules exhibited good convergence behavior, and 2) BSS tasks for several music sources with two microphones and three instrumental parts were evaluated successfully.

Online multiscale dynamic topic models

We propose an online topic model for sequentially analyzing the time evolution of topics in document collections. Topics naturally evolve with multiple timescales. For example, some words may be used consistently over one hundred years, while other words emerge and disappear over periods of a few days. Thus, in the proposed model, current topic-specific distributions over words are assumed to be generated based on the multiscale word distributions of the previous epoch. Considering both the long-timescale dependency as well as the short-timescale dependency yields a more robust model. We derive efficient online inference procedures based on a stochastic EM algorithm, in which the model is sequentially updated using newly obtained data; this means that past data are not required to make the inference. We demonstrate the effectiveness of the proposed method in terms of predictive performance and computational efficiency by examining collections of real documents with timestamps.

Probabilistic latent semantic visualization: topic model for visualizing documents

We propose a visualization method based on a topic model for discrete data such as documents. Unlike conventional visualization methods based on pairwise distances such as multi-dimensional scaling, we consider a mapping from the visualization space into the space of documents as a generative process of documents. In the model, both documents and topics are assumed to have latent coordinates in a two- or three-dimensional Euclidean space, or visualization space. The topic proportions of a document are determined by the distances between the document and the topics in the visualization space, and each word is drawn from one of the topics according to its topic proportions. A visualization, i.e. latent coordinates of documents, can be obtained by fitting the model to a given set of documents using the EM algorithm, resulting in documents with similar topics being embedded close together. We demonstrate the effectiveness of the proposed model by visualizing document and movie data sets, and quantitatively compare it with conventional visualization methods.