Shogo Nagasaka

Semiotic prediction of driving behavior using unsupervised double articulation analyzer

In this paper, we propose a novel semiotic prediction method for driving behavior based on double articulation structure. It has been reported that predicting driving behavior from its multivariate time series behavior data by using machine learning methods, e.g., hybrid dynamical system, hidden Markov model and Gaussian mixture model, is difficult because a drivers behavior is affected by various contextual information. To overcome this problem, we assume that contextual information has a double articulation structure and develop a novel semiotic prediction method by extending nonparametric Bayesian unsupervised morphological analyzer. Effectiveness of our prediction method was evaluated using synthetic data and real driving data. In these experiments, the proposed method achieved long-term prediction 2-6 times longer than some conventional methods.

Double articulation analyzer for unsegmented human motion using Pitman-Yor language model and infinite hidden Markov model

We propose an unsupervised double articulation analyzer for human motion data. Double articulation is a two-layered hierarchical structure underlying in natural language, human motion and other natural data produced by human. A double articulation analyzer estimates the hidden structure of observed data by segmenting and chunking target data. We develop a double articulation analyzer by using a sticky hierarchical Dirichlet process HMM (sticky HDP-HMM), a nonparametric Bayesian model, and an unsupervised morphological analysis based on nested Pitman-Yor language model which can chunk given documents without any dictionaries. We conducted an experiment to evaluate this method. The proposed method could extract unit motions from unsegmented human motion data by analyzing hidden double articulation structure.

Online learning of concepts and words using multimodal LDA and hierarchical Pitman-Yor Language Model

In this paper, we propose an online algorithm for multimodal categorization based on the autonomously acquired multimodal information and partial words given by human users. For multimodal concept formation, multimodal latent Dirichlet allocation (MLDA) using Gibbs sampling is extended to an online version. We introduce a particle filter, which significantly improve the performance of the online MLDA, to keep tracking good models among various models with different parameters. We also introduce an unsupervised word segmentation method based on hierarchical Pitman-Yor Language Model (HPYLM). Since the HPYLM requires no predefined lexicon, we can make the robot system that learns concepts and words in completely unsupervised manner. The proposed algorithms are implemented on a real robot and tested using real everyday objects to show the validity of the proposed system.

Unsupervised drive topic finding from driving behavioral data

Continuous driving-behavioral data can be converted automatically into sequences of “drive topics” in natural language; for example, “gas pedal operating,” “high-speed cruise,” then “stopping and standing still with brakes on.” In regard to developing advanced driver-assistance systems (ADASs), various methods for recognizing driver behavior have been proposed. Most of these methods employ a supervised approach based on human tags. Unfortunately, preparing complete annotations is practically impossible with massive driving-behavioral data because of the great variety of driving scenes. To overcome that difficulty, in this study, a double articulation analyzer (DAA) is used to segment continuous driving-behavioral data into sequences of discrete driving scenes. Thereafter, latent Dirichlet allocation (LDA) is used for clustering the driving scenes into a small number of so-called “drive topics” according to emergence frequency of physical features observed in the scenes. Because both DAA and LDA are unsupervised methods, they achieve data-driven scene segmentation and drive topic estimation without human tags. Labels of the extracted drive topics are also determined automatically by using distributions of the physical behavioral features included in each drive topic. The proposed framework therefore translates the output of sensors monitoring the driver and the driving environment into natural language. Efficiency of proposed method is evaluated by using a massive data set of driving behavior, including 90 drives for more than 78 hours over 3700km in total.

Contextual scene segmentation of driving behavior based on double articulation analyzer

Various advanced driver assistance systems (ADASs) have recently been developed, such as Adaptive Cruise Control and Precrash Safety System. However, most ADASs can operate in only some driving situations because of the difficulty of recognizing contextual information. For closer cooperation between a driver and vehicle, the vehicle should recognize a wider range of situations, similar to that recognized by the driver, and assist the driver with appropriate timing. In this paper, we assumed a double articulation structure in driving behavior data and segmented driving behavior into meaningful chunks for driving scene recognition in a similar manner to natural language processing (NLP). A double articulation analyzer translated the driving behavior into meaningless manemes, which are the smallest units of the driving behavior just like phonemes in NLP, and from them it constructed navemes, which are meaningful chunks of driving behavior just like morphemes. As a result of this two-phase analysis, we found that driving chunks equivalent to language words were closer to the complicated or contextual driving scene segmentation produced by human recognition.

Mutual learning of an object concept and language model based on MLDA and NPYLM

Humans develop their concept of an object by classifying it into a category, and acquire language by interacting with others at the same time. Thus, the meaning of a word can be learnt by connecting the recognized word and concept. We consider such an ability to be important in allowing robots to flexibly develop their knowledge of language and concepts. Accordingly, we propose a method that enables robots to acquire such knowledge. The object concept is formed by classifying multimodal information acquired from objects, and the language model is acquired from human speech describing object features. We propose a stochastic model of language and concepts, and knowledge is learnt by estimating the model parameters. The important point is that language and concepts are interdependent. There is a high probability that the same words will be uttered to objects in the same category. Similarly, objects to which the same words are uttered are highly likely to have the same features. Using this relation, the accuracy of both speech recognition and object classification can be improved by the proposed method. However, it is difficult to directly estimate the parameters of the proposed model, because there are many parameters that are required. Therefore, we approximate the proposed model, and estimate its parameters using a nested Pitman-Yor language model and multimodal latent Dirichlet allocation to acquire the language and concept, respectively.

Unsupervised Hierarchical Modeling of Driving Behavior and Prediction of Contextual Changing Points

An unsupervised learning method, called double articulation analyzer with temporal prediction (DAA-TP), is proposed on the basis of the original DAA model. The method will enable future advanced driving assistance systems to determine driving context and predict possible scenarios of driving behavior by segmenting and modeling incoming driving-behavior time series data. In previous studies, we applied the DAA model to driving-behavior data and argued that contextual changing points can be estimated as changing points of chunks. A sequence prediction method, which predicts the next hidden state sequence, was also proposed in a previous study. However, the original DAA model does not model the duration of chunks of driving behavior and is not able to do a temporal prediction of the scenarios. Our DAA-TP method explicitly models the duration of chunks of driving behavior on the assumption that driving-behavior data have a two-layered hierarchical structure, i.e., double articulation structure. For this purpose, the hierarchical Dirichlet process hidden semi-Markov model is used for explicitly modeling the duration of segments of driving-behavior data. A Poisson distribution is also used to model the duration distribution of driving-behavior segments. The duration distribution of chunks of driving-behavior data is also theoretically calculated using the reproductive property of the Poisson distribution. We also propose a calculation method for obtaining the probability distribution of the remaining duration of current driving words as a mixture of Poisson distribution with a theoretical approximation for unobserved driving words. This method can calculate the posterior probability distribution of the next termination time of chunks by explicitly modeling all probable chunking results for observed data. The DAA-TP was applied to a synthetic data set having a double articulation structure to evaluate its model consistency. To evaluate the effectiveness of DAA-TP, we applied it to a driving-behavior data set recorded at actual factory circuits. The DAA-TP could predict the next termination time of chunks more accurately than the compared methods. We also report the qualitative results for understanding the potential capability of DAA-TP.

Automatic drive annotation via multimodal latent topic model

Time-series driving behavioral data and image sequences captured with car-mounted video cameras can be annotated automatically in natural language, for example, “in a traffic jam,” “leading vehicle is a truck,” or “there are three and more lanes.” Various driving support systems nowadays have been developed for safe and comfortable driving. To develop more effective driving assist systems, abstractive recognition of driving situation performed just like a human driver is important in order to achieve fully cooperative driving between the driver and vehicle. To achieve human-like annotation of driving behavioral data and image sequences, we first divided continuous driving behavioral data into discrete symbols that represent driving situations. Then, using multimodal latent Dirichlet allocation, latent driving topics laid on each driving situation were estimated as a relation model among driving behavioral data, image sequences, and human-annotated tags. Finally, automatic annotation of the behavioral data and image sequences can be achieved by calculating the predictive distribution of the annotations via estimated latent-driving topics. The proposed method intuitively annotated more than 50,000 pieces of frame data, including urban road and expressway data. The effectiveness of the estimated drive topics was also evaluated by analyzing the performances of driving-situation classification. The topics represented the drive context efficiently, i.e., the drive topics lead to a 95% lower-dimensional feature space and 6% higher accuracy compared with a high-dimensional raw-feature space. Moreover, the drive topics achieved performance almost equivalent performance texpressway datao human annotators, especially in classifying traffic jams and the number of lanes.

Nonparametric Bayesian Double Articulation Analyzer for Direct Language Acquisition From Continuous Speech Signals

Human infants can discover words directly from unsegmented speech signals without any explicitly labeled data. Current machine learning methods cannot efficiently estimate language model (LM) and acoustic model (AM) and discover words directly from continuous human speech signals in an unsupervised manner. To solve this problem, we propose an integrative generative model that combines an LM and an AM into a single generative model called the hierarchical Dirichlet process hidden LM (HDP-HLM). The HDP-HLM is obtained by extending the hierarchical Dirichlet process hidden semi-Markov model (HDP-HSMM) proposed by Johnson et al. An inference procedure for the HDP-HLM is derived using the blocked Gibbs sampler originally proposed for the HDP-HSMM. This procedure enables the simultaneous and direct inference of LM and AM from continuous speech signals. Based on the HDP-HLM and its inference procedure, we develop a novel machine learning method called nonparametric Bayesian double articulation analyzer (NPB-DAA) that can directly acquire LM and AM from observed continuous speech signals. By assuming HDP-HLM as a generative model of observed time series data, and by inferring latent variables of the model, the method can analyze latent double articulation structure, i.e., hierarchically organized latent words and phonemes, of the data in an unsupervised manner. We also carried out two evaluation experiments using synthetic data and actual human continuous speech signals representing Japanese vowel sequences. In the word acquisition and phoneme categorization tasks, the NPB-DAA outperformed a conventional double articulation analyzer and baseline automatic speech recognition system whose AM was trained in a supervised manner. The main contributions of this paper are as follows: 1) we develop a probabilistic generative model that integrates LM and AM, i.e., HDP-HLM; 2) we derive an inference method for this, and propose the NPB-DAA; and 3) we show that the NPB-DAA can discover words directly from continuous human speech signals in an unsupervised manner.

Sequence Prediction of Driving Behavior Using Double Articulation Analyzer

A sequence prediction method for driving behavior data is proposed in this paper. The proposed method can predict a longer latent state sequence of driving behavior data than conventional sequence prediction methods. The proposed method is derived by focusing on the double articulation structure latently embedded in driving behavior data. The double articulation structure is a two-layer hierarchical structure originally found in spoken language, i.e., a sentence is a sequence of words and a word is a sequence of letters. Analogously, we assume that driving behavior data comprise a sequence of driving words and a driving word is a sequence of driving letters. The sequence prediction method is obtained by extending a nonparametric Bayesian unsupervised morphological analyzer using a nested Pitman-Yor language model (NPYLM), which was originally proposed in the natural language processing field. This extension allows the proposed method to analyze incomplete sequences of latent states of driving behavior and to predict subsequent latent states on the basis of a maximum a posteriori criterion. The extension requires a marginalization technique over an infinite number of possible driving words. We derived such a technique on the basis of several characteristics of the NPYLM. We evaluated this proposed sequence prediction method using three types of data: 1) synthetic data; 2) data from test drives around a driving course at a factory; and 3) data from drives on a public thoroughfare. The results showed that the proposed method made better long-term predictions than did the previous methods.