Welly Naptali

Topic-Dependent-Class-Based

A topic-dependent-class (TDC)-based n-gram language model (LM) is a topic-based LM that employs a semantic extraction method to reveal latent topic information extracted from noun-noun relations. A topic of a given word sequence is decided on the basis of most frequently occuring (weighted) noun classes in the context history through voting. Our previous work (W. Naptali, M. Tsuchiya, and S. Seiichi, “Topic-dependent language model with voting on noun history,”ACM Trans. Asian Language Information Processing (TALIP), vol. 9, no. 2, pp. 1-31, 2010) has shown that in terms of perplexity, TDCs outperform several state-of-the-art baselines, i.e., a word-based or class-based n-gram LM and their interpolation, a cache-based LM, an n-gram-based topic-dependent LM, and a Latent Dirichlet Allocation (LDA)-based topic-dependent LM. This study is a follow up of our previous work and there are three key differences. First, we improve TDCs by employing soft-clustering and/or soft-voting techniques, which solve data shrinking problems and make TDCs independent of the word-based n-gram, in the training and/or test phases. Second, for further improvement, we incorporate a cache-based LM through unigram scaling, because the TDC and cache-based LM capture different properties of the language. Finally, we provide an evaluation in terms of the word error rate (WER) and an analysis of the automatic speech recognition (ASR) rescoring task. Experiments performed on the Wall Street Journal and the Mainichi Shimbun (a Japanese newspaper) demonstrate that the TDC LM improves both perplexity and the WER. The perplexity reduction is up to 25.1% relative on the English corpus and 25.7% relative on the Japanese corpus. Furthermore, the greatest reduction in the WER is 15.2% relative to the English ASR and 24.3 relative to the Japanese ASR, as compared to the baseline.

n

Language models (LMs) are an important field of study in automatic speech recognition (ASR) systems. LM helps acoustic models find the corresponding word sequence of a given speech signal. Without it, ASR systems would not understand the language and it would be hard to find the correct word sequence. During the past few years, researchers have tried to incorporate long-range dependencies into statistical word-based n-gram LMs. One of these long-range dependencies is topic. Unlike words, topic is unobservable. Thus, it is required to find the meanings behind the words to get into the topic. This research is based on the belief that nouns contain topic information. We propose a new approach for a topic-dependent LM, where the topic is decided in an unsupervised manner. Latent Semantic Analysis (LSA) is employed to reveal hidden (latent) relations among nouns in the context words. To decide the topic of an event, a fixed size word history sequence (window) is observed, and voting is then carried out based on noun class occurrences weighted by a confidence measure. Experiments were conducted on an English corpus and a Japanese corpus: The Wall Street Journal corpus and Mainichi Shimbun (Japanese newspaper) corpus. The results show that our proposed method gives better perplexity than the comparative baselines, including a word-based/class-based n-gram LM, their interpolated LM, a cache-based LM, a topic-dependent LM based on n-gram, and a topic-dependent LM based on Latent Dirichlet Allocation (LDA). The n-best list rescoring was conducted to validate its application in ASR systems.

-Gram Language Model

A topic dependent class (TDC) language model (LM) is a topic-based LM that uses a semantic extraction method to reveal latent topic information from noun-document relation. Then a clustering for a given context is performed to define topics. Finally, a fixed window of word history is observed to decide the topic of the current event through voting in online manner. Previously, we have shown that TDC overperforms several state-of-the-art baselines in terms of perplexity. In this paper we evaluate TDC on automatic speech recognition experiment (ASR) for rescoring task. Experiments on read speech Wall Street Journal (English ASR system) and Mainichi Shimbun (Japanese ASR system) show that TDC LM improves both perplexity and word-error-rate (WER). The result shows that the proposed model gives improvements 3.0% relative on perplexity and 15.2% relative on WER for English ASR system, and 16.4% relative on perplexity and 24.3% relative on WER for Japanese ASR system.

Topic-Dependent Language Model with Voting on Noun History

This paper investigates matrix representation in latent semantic analysis (LSA) framework for a language model. In LSA, word-document matrix is usually used to represent a corpus. However, this matrix ignores word order in the sentence. We propose several word co-occurrence matrices that keep word order to use in LSA. To support this matrix, we define a context dependent class (CDC) language model, which distinguishes classes according to their context in the sentences. Experiments on Wall Street Journal (WSJ) corpus show that the proposed method achieves better performance than the original LSA with word-document matrix.