Yuki Yamaguchi

Audio-visual speech recognition using deep learning

Audio-visual speech recognition (AVSR) system is thought to be one of the most promising solutions for reliable speech recognition, particularly when the audio is corrupted by noise. However, cautious selection of sensory features is crucial for attaining high recognition performance. In the machine-learning community, deep learning approaches have recently attracted increasing attention because deep neural networks can effectively extract robust latent features that enable various recognition algorithms to demonstrate revolutionary generalization capabilities under diverse application conditions. This study introduces a connectionist-hidden Markov model (HMM) system for noise-robust AVSR. First, a deep denoising autoencoder is utilized for acquiring noise-robust audio features. By preparing the training data for the network with pairs of consecutive multiple steps of deteriorated audio features and the corresponding clean features, the network is trained to output denoised audio features from the corresponding features deteriorated by noise. Second, a convolutional neural network (CNN) is utilized to extract visual features from raw mouth area images. By preparing the training data for the CNN as pairs of raw images and the corresponding phoneme label outputs, the network is trained to predict phoneme labels from the corresponding mouth area input images. Finally, a multi-stream HMM (MSHMM) is applied for integrating the acquired audio and visual HMMs independently trained with the respective features. By comparing the cases when normal and denoised mel-frequency cepstral coefficients (MFCCs) are utilized as audio features to the HMM, our unimodal isolated word recognition results demonstrate that approximately 65 % word recognition rate gain is attained with denoised MFCCs under 10 dB signal-to-noise-ratio (SNR) for the audio signal input. Moreover, our multimodal isolated word recognition results utilizing MSHMM with denoised MFCCs and acquired visual features demonstrate that an additional word recognition rate gain is attained for the SNR conditions below 10 dB.

Tool-Body Assimilation Model Based on Body Babbling and Neurodynamical System

We propose the new method of tool use with a tool-body assimilation model based on body babbling and a neurodynamical system for robots to use tools. Almost all existing studies for robots to use tools require predetermined motions and tool features; the motion patterns are limited and the robots cannot use novel tools. Other studies fully search for all available parameters for novel tools, but this leads to massive amounts of calculations. To solve these problems, we took the following approach: we used a humanoid robot model to generate random motions based on human body babbling. These rich motion experiences were used to train recurrent and deep neural networks for modeling a body image. Tool features were self-organized in parametric bias, modulating the body image according to the tool in use. Finally, we designed a neural network for the robot to generate motion only from the target image. Experiments were conducted with multiple tools for manipulating a cylindrical target object. The results show that the tool-body assimilation model is capable of motion generation.

Multiple-channel wavelength conversions in a photonic crystal cavity.

We demonstrate multiple-channel wavelength conversions of second harmonic and sum frequency generations in a silicon carbide photonic crystal cavity. The cavity is designed to have multiple modes including a nanocavity mode and Fabry-Pérot modes. Multiple-channel wavelength conversions in the nanocavity and Fabry-Pérot modes are shown experimentally. Furthermore, we investigate the polarization characteristics of wavelength-converted light. The experimental results of the polarization are in good agreement with calculation.

Analysis of Q-factors of structural imperfections in triangular cross-section nanobeam photonic crystal cavities

We present comprehensive and quantitative analysis of the effect of structural imperfections on quality (Q)-factors in triangular cross-section nanobeam photonic crystal cavities. We investigated statistically the optical losses due to the various imperfections in the air holes’ positions, radii, alignments, and surface roughness, among other factors. It is revealed that the Q-factor decreases significantly from an ideally designed value due to such imperfections, with the main influence being the asymmetric alignment of the air hole line relative to the center of the nanobeam in the currently used fabrication process. Our analysis provides important information for achieving higher Q-factors in the cavities.

Tool-body assimilation model using a neuro-dynamical system for acquiring representation of tool function and motion

In this paper, we propose a tool-body assimilation model that implements a multiple time-scales recurrent neural network (MTRNN). Our model allows a robot to acquire the representation of a tool function and the required motion without having any prior knowledge of the tool. It is composed of five modules: image feature extraction, body model, tool dynamics feature, tool recognition, and motion recognition. Self-organizing maps (SOM) are used for image feature extraction from raw images. The MTRNN is used for body model learning. Parametric bias (PB) nodes are used to learn tool dynamic features. The PB nodes are attached to the neurons of the MTRNN to modulate the body model. A hierarchical neural network (HNN) is implemented for tool and motion recognition. Experiments were conducted using OpenHRP3, a robotics simulator, with multiple tools. The results show that the tool-body assimilation model is capable of recognizing tools, including those having an unlearned shape, and acquires the required motions accordingly.

Measurement of optical loss in nanophotonic waveguides using integrated cavities

Measurement of optical loss in nanophotonic waveguides is necessary for monitoring the properties of integrated photonic devices. We propose a simple method of measuring the optical loss using integrated nanocavities. It is shown theoretically that weak coupling between the waveguide and cavities leads to a direct estimation of the optical loss by measuring light radiated from the cavities. In addition, we experimentally demonstrate the optical loss in a fabricated photonic crystal waveguide. Our method gives not only a degree of freedom in real-time monitoring of the optical properties of nanophotonic structures, but it also can be used for various waveguide-based applications.

Learning and association of synaesthesia phenomenon using deep neural networks

Robots are required to process multimodal information because the information in the real world comes from various modal inputs. However, there exist only a few robots integrating multimodal information. Humans can recognize the environment effectively by cross-modal processing. We focus on modeling synaesthesia phenomenon known to be a cross-modal perception of humans. Recently, deep neural networks (DNNs) have gained more attention and successfully applied to process high-dimensional data composed not only of single modality but also of multimodal information. We introduced DNNs to construct multimodal association model which can reconstruct one modality from the other modality. Our model is composed of two DNNs: one for image compression and the other for audio-visual sequential learning. We tried to reproduce synaesthesia phenomenon by training our model with the multimodal data acquired from psychological experiment. Cross-modal association experiment showed that our model can reconstruct the same or similar images from sound as synaesthetes, those who experience synaesthesia. The analysis of middle layers of DNNs representing multimodal features implied that DNNs self-organized the difference of perception between individual synaesthetes.