Heeyoul Choi

Robust kernel Isomap

Isomap is one of widely used low-dimensional embedding methods, where geodesic distances on a weighted graph are incorporated with the classical scaling (metric multidimensional scaling). In this paper we pay our attention to two critical issues that were not considered in Isomap, such as: (1) generalization property (projection property); (2) topological stability. Then we present a robust kernel Isomap method, armed with such two properties. We present a method which relates the Isomap to Mercer kernel machines, so that the generalization property naturally emerges, through kernel principal component analysis. For topological stability, we investigate the network flow in a graph, providing a method for eliminating critical outliers. The useful behavior of the robust kernel Isomap is confirmed through numerical experiments with several data sets.

RNNDROP: A novel dropout for RNNS in ASR

Recently, recurrent neural networks (RNN) have achieved the state-of-the-art performance in several applications that deal with temporal data, e.g., speech recognition, handwriting recognition and machine translation. While the ability of handling long-term dependency in data is the key for the success of RNN, combating over-fitting in training the models is a critical issue for achieving the cutting-edge performance particularly when the depth and size of the network increase. To that end, there have been some attempts to apply the dropout, a popular regularization scheme for the feed-forward neural networks, to RNNs, but they do not perform as well as other regularization scheme such as weight noise injection. In this paper, we propose rnnDrop, a novel variant of the dropout tailored for RNNs. Unlike the existing methods where dropout is applied only to the non-recurrent connections, the proposed method applies dropout to the recurrent connections as well in such a way that RNNs generalize well. Our experiments show that rnnDrop is a better regularization method than others including weight noise injection. Namely, when deep bidirectional long short-term memory (LSTM) RNNs were trained with rnnDrop as acoustic models for phoneme and speech recognition, they significantly outperformed the current state-of-the-arts; we achieved the phoneme error rate of 16.29% on the TIMIT core test set for phoneme recognition and the word error rate of 5.53% on the Wall Street Journal (WSJ) dataset, dev93, for speech recognition, which are the best reported results on both of the datasets.

Kernel Isomap on Noisy Manifold

In the human brain, it is well known that perception is based on similarity rather than coordinates and it is carried out on the manifold of data set. Isomap (Tenenbaum et al., 2000) is one of widely-used low-dimensional embedding methods where approximate geodesic distance on a weighted graph is used in the framework of classical scaling (metric MDS). In this paper, we consider two critical issues missing in Isomap: (1) generalization property; (2) topological stability and present our robust kernel Isomap method, armed with such two properties. The useful behavior and validity of our robust kernel Isomap, is confirmed through numerical experiments with several data sets including real world data

A relative trust-region algorithm for independent component analysis

In this paper we present a method of parameter optimization, relative trust-region learning, where the trust-region method and the relative optimization [M. Zibulevsky, Blind source separation with relative Newton method, in: Proceedings of the ICA, Nara, Japan, 2003, pp. 897-902] are jointly exploited. The relative trust-region method finds a direction and a step size with the help of a quadratic model of the objective function (as in the conventional trust-region methods) and updates parameters in a multiplicative fashion (as in the relative optimization). We apply this relative trust-region learning method to the problem of independent component analysis (ICA), which leads to the relative TR-ICA algorithm which turns out to possess the equivariant property (as in the relative gradient) and to achieve faster convergence than the relative gradient and even Newton-type algorithms. Empirical comparisons with several existing ICA algorithms demonstrate the useful behavior of the relative TR-ICA algorithm, such as the equivariant property and fast convergence.

Learning α-integration with partially-labeled data

Sensory data integration is an important task in human brain for multimodal processing as well as in machine learning for multisensor processing. α-integration was proposed by Amari as a principled way of blending multiple positive measures (e.g., stochastic models in the form of probability distributions), providing an optimal integration in the sense of minimizing the α-divergence. It also encompasses existing integration methods as its special case, e.g., weighted average and exponential mixture. In α-integration, the value of α determines the characteristics of the integration and the weight vector w assigns the degree of importance to each measure. In most of the existing work, however, α and w are given in advance rather than learned. In this paper we present two algorithms, for learning α and w from data when only a few integrated target values are available. Numerical experiments on synthetic as well as real-world data confirm the proposed methods effectiveness.

Gesture Recognition Based on Manifold Learning

Current feature-based gesture recognition systems use human-chosen features to perform recognition. Effective features for classification can also be automatically learned and chosen by the computer. In other recognition domains, such as face recognition, manifold learning methods have been found to be good nonlinear feature extractors. Few manifold learning algorithms, however, have been applied to gesture recognition. Current manifold learning techniques focus only on spatial information, making them undesirable for use in the domain of gesture recognition where stroke timing data can provide helpful insight into the recognition of hand-drawn symbols. In this paper, we develop a new algorithm for multi-stroke gesture recognition, which integrates timing data into a manifold learning algorithm based on a kernel Isomap. Experimental results show it to perform better than traditional human-chosen feature-based systems.

Relative Gradient Learning for Independent Subspace Analysis

Independent subspace analysis (ISA) is a generalization of independent component analysis (ICA), where multidimensional ICA is incorporated with the idea of invariant feature subspaces, allowing components in the same subspace to be dependent, but requiring independence between feature subspaces. In this paper we present a relative gradient algorithm for ISA, derived in the framework of the relative optimization as well as in a direct manner. Empirical comparison with the gradient ISA algorithm, shows that the relative gradient ISA algorithm achieves faster convergence, compared to the conventional gradient algorithm.

Fast and accurate retinal vasculature tracing and kernel-Isomap-based feature selection

The blood vessels in the retina have a characteristic radiating pattern, while there exists a significant variation dependent on the individual and/or medical condition. Extracting the geometric properties of these blood vessels have several important applications, such as biometrics (for identification) and medical diagnosis. In this paper, we will focus on biometric applications. For this, we propose a fast and accurate algorithm for tracing the blood vessels, and compare several candidate summary features based on the tracing results. Existing tracing algorithms based on a detailed analysis of the image can be too slow to quickly process a large volume of retinal images in real time (e.g., at a security check point). In order to select good features that can be extracted from the traces, we used kernel Isomap to test the distance between different retinal images as projected onto their respective feature spaces. We tested the following feature set: (1) angle among branches, (2) the number of fiber based on distance, (3) distance between branches, and (4) inner product among branches. Our results indicate that features 3 and 4 are prime candidates for use in fast, realtime biometric tasks. We expect our method to lead to fast and accurate biometric systems based on retinal images.

Manifold alpha-integration

Manifold learning has been successfully used for finding dominant factors (low-dimensional manifold) in a high-dimensional data set. However, most existing manifold learning algorithms only consider one manifold based on one dissimilarity matrix. For utilizing multiple manifolds, a key question is how different pieces of information can be integrated when multiple measurements are available. Amari proposed a-integration for stochastic model integration, which is a generalized averaging method that includes as a special case arithmetic, geometric, and harmonic averages. In this paper, we propose a new generalized manifold integration algorithm equipped with a-integration, manifold α-integration (MAI). Interestingly, MAI can be shown to be a generalization of other integration methods (that may or may not use manifolds) like kernel fusion or mixture of random walk. Our experimental results also confirm that integration of multiple sources of information on individual manifolds is superior to the use of individual manifolds separately, in tasks including classification and sensorimotor integration.

Data visualization for asymmetric relations

Most 2D visualization methods based on multidimensional scaling (MDS) and self-organizing maps (SOMs) use a symmetric distance matrix to represent and visualize object relationships in a data set. In many real-world applications, however, raw data such as a world-trade data are best captured as an asymmetric proximity matrix. Such asymmetric matrices cannot be perfectly represented by most previous methods. To handle such an intrinsic limitation, in this paper, we propose a dynamic learning for metric representations of asymmetric proximity data to better understand the data. The proposed learning generates two representations (maps) with the row vectors (sending or exporting) and column vectors (receiving or importing) of the matrix, respectively. To better present the patterns, we supplement the maps with two analysis tools: cluster analysis and distance analysis, which connect and compare the different patterns from the different maps. Experiment results using three real world data sets confirm that the proposed learning method is useful to understand asymmetric proximity data.