Nobuo Suematsu

A multiagent reinforcement learning algorithm using extended optimal response

Stochastic games provides a theoretical framework to multiagent reinforcement learning. Based on the framework, a multiagent reinforcement learning algorithm for zero-sum stochastic games was proposed by Littman and it was extended to general-sum games by Hu and Wellman. Given a stochastic game, if all agents learn with their algorithm, we can expect that the policies of the agents converge to a Nash equilibrium. However, agents with their algorithm always try to converge to a Nash equilibrium independent of the policies used by the other agents. In addition, in case there are multiple Nash equilibria, agents must agree on the equilibrium where they want to reach. Thus, their algorithm lacks adaptability in a sense. In this paper, we propose a multiagent reinforcement learning algorithm. The algorithm uses the extended optimal response which we introduce in this paper. It will converge to a Nash equilibrium when other agents are adaptable, otherwise it will make an optimal response. We also provide some empirical results in three simple stochastic games, which show that the algorithm can realize what we intend.

Image Segmentation Using a Spatially Correlated Mixture Model with Gaussian Process Priors

Finite mixture modeling has been widely used for image segmentation. However, since it takes no account of the spatial correlation among pixels in its standard form, its segmentation accuracy can be heavily deteriorated by noise in images. To improve segmentation accuracy in noisy images, the spatially variant finite mixture model has been proposed, in which a Markov Random Filed (MRF) is used as the prior for the mixing proportions and its parameters are estimated using the Expectation-Maximization (EM) algorithm based on the maximum a posteriori (MAP) criterion. In this paper, we propose a spatially correlated mixture model in which the mixing proportions are governed by a set of underlying functions whose common prior distribution is a Gaussian process. The spatial correlation can be expressed with a Gaussian process easily and flexibly. Given an image, the underlying functions are estimated by using a quasi EM algorithm and used to segment the image. The effectiveness of the proposed technique is demonstrated by an experiment with synthetic images.

A new shape descriptor based on an angular-linear probability distribution

Motivated by the increased consideration of probability distributions as local descriptors of shape, we propose a local descriptor based on a bivariate circular distribution. Although some bivariate circular distributions are difficult to compute, our descriptor is computationally feasible because it is a generalization of the mixture of von Mises distributions. Using various shapes formed by line drawings, we show that our descriptor is more effective in shape retrieval than several conventional local descriptors.

Hierarchical hidden conditional random fields for information extraction

Hidden Markov Models (HMMs) are very popular generative models for time series data. Recent work, however, has shown that for many tasks Conditional Random Fields (CRFs), a type of discriminative model, perform better than HMMs. Information extraction is the task of automatically extracting instances of specified classes or relations from text. A method for information extraction using Hierarchical Hidden Markov Models (HHMMs) has already been proposed. HHMMs, a generalization of HMMs, are generative models with a hierarchical state structure. In previous research, we developed the Hierarchical Hidden Conditional Random Field (HHCRF), a discriminative model corresponding to HHMMs. In this paper, we propose information extraction using HHCRFs, and then compare the performance of HHMMs and HHCRFs through an experiment.

A nonparametric Bayesian approach to time series alignment

We propose a nonparametric Bayesian approach to time series alignment. Time series alignment is a technique often required when we analyze a set of time series in which there exists a typical structural pattern common to all the time series. Such a set of time series is usually obtained by repeated measurements of a biological, chemical or physical process. In time series alignment, we required to estimate a common shape function, which describes a common structural patter shared among a set of time series, and time transformation (time warping) functions, each of which represents time shifts involved in individual time series. In our approach, the common shape function and the time transformation functions are modeled nonparametrically by using Gaussian process priors. We introduce an effective Markov Chain Monte Carlo algorithm and it enables a fully Bayesian analysis of time series alignment.

Real world map building and localization through continuous sonar sensing and clustering

Map building in the real world using sonars poses two challenges: 1) noisy sonar readings which need sophisticated preprocessing; and 2) complicated floor plans which make it difficult to extract key features in the environment. We have developed a continuous sonar sensing and clustering method to solve the problems. The method allows us not only to discard sonar readings irrelevant for localization but also to extract key environment features that can be traced for some period of time. An important advantage of our method is that we do not need to register key environment features (i.e. landmark patterns) in advance. We have successfully implemented and tested a map building and localization system using the method. We report on our experiment conducted using a physical robot in an unmodified large scale environment.

A sampling method for processing contours drawn with an uncertain stroke order and number

Although there are several effective methods for placing sample points on contours drawn with a certain stroke order and number, little attention has been paid to methods for placing sample points on contours with an uncertain stroke order and number. In this paper, we place sample points appropriately on such contours using an optimization method. The underlying idea is to take into account the degree in graph theory and edge length in geometry for the optimization. Using a dataset of line drawings, we confirm that our method, provides an appropriate placement of sample points in terms of shape retrieval.

Finding the most likely upper level state sequence for hierarchical HMMs

Computing the most likely state sequence from an observation sequence is an important problem with many applications. The generalized Viterbi algorithm, a direct extension of the Viterbi algorithm for hidden Markov models (HMMs), has been used to find the most likely state sequence for hierarchical HMMs. However, the generalized Viterbi algorithm finds the most likely whole level state sequence rather than the most likely upper level state sequence. In this paper, we propose a marginalized Viterbi algorithm, which finds the most likely upper level state sequence by marginalizing lower level state sequences. We show experimentally that the marginalized Viterbi algorithm is more accurate than the generalized Viterbi algorithm in terms of upper level state sequence estimation.

Marginalized Viterbi algorithm for hierarchical hidden Markov models

The generalized Viterbi algorithm, a direct extension of the Viterbi algorithm for hidden Markov models (HMMs), has been used to find the most likely state sequence for hierarchical HMMs. However, the generalized Viterbi algorithm finds the most likely whole level state sequence rather than the most likely upper level state sequence. In this paper, we propose a marginalized Viterbi algorithm, which finds the most likely upper level state sequence by marginalizing lower level state sequences. We show experimentally that the marginalized Viterbi algorithm is more accurate than the generalized Viterbi algorithm in terms of upper level state sequence estimation.

Component Reduction for Hierarchical Mixture Model Construction

The mixture modeling framework is widely used in many applications. In this paper, we propose a component reductiontechnique, that collapses a mixture model into a mixture with fewer components. For fitting a mixture model to data, the EM (Expectation-Maximization) algorithm is usually used. Our algorithm is derived by extending mixture model learning using the EM-algorithm. In this extension, a difficulty arises from the fact that some crucial quantities cannot be evaluated analytically. We overcome this difficulty by introducing an effective approximation. The effectiveness of our algorithm is demonstrated by applying it to a simple synthetic component reduction task and a phoneme clustering problem.