Sung Joo Lee

Deep neural network using trainable activation functions.

This paper proposes trainable activation functions for deep neural network (DNN). A DNN is a feed-forward neural network composed of more than one hidden nonlinear layer. It is characterized by a set of weight matrices, bias vectors, and a nonlinear activation function. In model parameter training, weight matrices and bias vectors are updated using an error back-propagation algorithm but activation functions is not. It is just fixed empirically. Many rectifier-type nonlinear functions have been proposed as activation functions, but the best nonlinear functions for any given task domain remain unknown. In order to address the issue, we propose a trainable activation function. In the proposed approach, conventional nonlinear activation functions were approximated for a Taylor series, and the coefficients were retrained simultaneously with other parameters. The effectiveness of the proposed approach was evaluated for MNIST handwritten digit recognition domain.

Deep neural network based acoustic model parameter reduction using manifold regularized low rank matrix factorization

In this paper, we propose a deep neural network (DNN) model parameter reduction based on manifold regularized low rank matrix factorization to reduce the computational complexity of acoustic model for low resource embedded devices. One of the most common DNN model parameter reduction techniques is truncated singular value decomposition (TSVD). TSVD reduces the number of parameters by approximating a target matrix with a low rank one in terms of minimizing the Euclidean norm. In this work, we questioned whether the Euclidean norm is appropriate as objective function to factorize DNN matrices because DNN is known to learn nonlinear manifold of acoustic features. Therefore, in order to exploit the manifold structure for robust parameter reduction, we propose manifold regularized matrix factorization approach. The proposed method was evaluated on TIMIT phone recognition domain.

Phonetic state relation graph regularized deep neural network for robust acoustic model

In this paper, we propose a phonetic state relation graph regularized Deep Neural Network (DNN) for a robust acoustic model. A DNN-based acoustic model is trained in terms of minimizing a cost function that is usually penalized by regularizations. Regularization generally reflects prior knowledge that plays a role in constraining the model parameter space. In DNN-based acoustic models, various regularizations have been proposed to improve robustness. However, most approaches do not handle speech generation knowledge even if this process is the most fundamental prior. For example, l1 and l2-norm regularizations are equivalent to set Gaussian prior and Laplacian prior to model parameters respectively. This means that any speech signal specific knowledge is not used for regularization. Manifold-based regularization exploits the local linear structure of observed acoustic features, which are simply realization of the speech generation process. Therefore, to incorporate prior knowledge of speech generation into regularization, we propose a phonetic state relation graph based approach. This method was evaluated on the TIMIT phone recognition domain. The results showed that it reduced the phone error rate from 20.8% to 20.3% under the same conditions.

A useful feature-engineering approach for a LVCSR system based on CD-DNN-HMM algorithm

In this paper, we propose a useful feature-engineering approach for Context-Dependent Deep-Neural-Network Hidden-Markov-Model (CD-DNN-HMM) based Large-Vocabulary-Continuous-Speech-Recognition (LVCSR) systems. The speech recognition performance of a LVCSR system is improved from two feature-engineering perspectives. The first performance improvement is achieved by adopting the intra/inter-frame feature subsets when the Gaussian-Mixture-Model (GMM) HMMs for the HMM state-level alignment are built. And the second performance gain is then followed with the additional features augmenting the front-end of the DNN. We evaluate the effectiveness of our feature-engineering approach under a series of Korean speech recognition tasks (isolated single-syllable recognition with a medium-sized speech corpus and conversational speech recognition with a large-sized database) using the Kaldi speech recognition toolkit. The results show that the proposed feature-engineering approach outperforms the traditional Mel Frequency Cepstral Coefficient (MFCCs) GMM + Mel-frequency filter-bank output DNN method.