Yidong Chen

Lattice-to-sequence attentional Neural Machine Translation models

Abstract The dominant Neural Machine Translation (NMT) models usually resort to word-level modeling to embed input sentences into semantic space. However, it may not be optimal for the encoder modeling of NMT, especially for languages where tokenizations are usually ambiguous: On one hand, there may be tokenization errors which may negatively affect the encoder modeling of NMT. On the other hand, the optimal tokenization granularity is unclear for NMT. In this paper, we propose lattice-to-sequence attentional NMT models, which generalize the standard Recurrent Neural Network (RNN) encoders to lattice topology. Specifically, they take as input a word lattice which compactly encodes many tokenization alternatives, and learn to generate the hidden state for the current step from multiple inputs and hidden states in previous steps. Compared with the standard RNN encoder, the proposed encoders not only alleviate the negative impact of tokenization errors but are more expressive and flexible as well for encoding the meaning of input sentences. Experimental results on both Chinese–English and Japanese–English translations demonstrate the effectiveness of our models.

Co-training for Implicit Discourse Relation Recognition Based on Manual and Distributed Features

Implicit discourse relation recognition aims to discover the semantic relation between two sentences where the discourse connective is absent. Due to the lack of labeled data, previous work tries to generate additional training data automatically by removing discourse connectives from explicit discourse relation instances. However, using these artificial data indiscriminately has been proven to degrade the performance of implicit discourse relation recognition. To address this problem, we propose a co-training approach based on manual features and distributed features, which identifies useful instances from these artificial data to enlarge the labeled data. In addition, the distributed features are learned via recursive autoencoder based approaches, capable of capturing to some extent the semantics of sentences which is valuable for implicit discourse relation recognition. Experiment results on both the PDTB and CDTB data sets indicate that: (1) The learned distributed features are complementary to the manual features, and thus suitable for co-training. (2) Our proposed co-training approach can use these artificial data effectively, and significantly outperforms the baselines.

An SNN-Based Semantic Role Labeling Model with Its Network Parameters Optimized Using an Improved PSO Algorithm

Semantic role labeling (SRL) is a fundamental task in natural language processing to find a sentence-level semantic representation. The semantic role labeling procedure can be viewed as a process of competition between many order parameters, in which the strongest order parameter will win by competition and the desired pattern will be recognized. To realize the above-mentioned integrative SRL, we use synergetic neural network (SNN). Since the network parameters of SNN directly influence the synergetic recognition performance, it is important to optimize the parameters. In this paper, we propose an improved particle swarm optimization (PSO) algorithm based on log-linear model and use it to effectively determine the network parameters. Our contributions are two-folds: firstly, a log-linear model is introduced to PSO algorithm which can effectively make use of the advantages of a variety of different knowledge sources, and enhance the decision making ability of the model. Secondly, we propose an improved SNN model based on the improved PSO and show its effectiveness in the SRL task. The experimental results show that the proposed model has a higher performance for semantic role labeling with more powerful global exploration ability and faster convergence speed, and indicate that the proposed model has a promising future for other natural language processing tasks.

Leveraging bilingually-constrained synthetic data via multi-task neural networks for implicit discourse relation recognition

Recognizing implicit discourse relations is an important but challenging task in discourse understanding. To alleviate the shortage of labeled data, previous work automatically generates synthetic implicit data (SynData) as additional training data, by removing connectives from explicit discourse instances. Although SynData has been proven useful for implicit discourse relation recognition, it also has the meaning shift problem and the domain problem. In this paper, we first propose to use bilingually-constrained synthetic implicit data (BiSynData) to enrich the training data, which can alleviate the drawbacks of SynData. Our BiSynData is constructed from a bilingual sentence-aligned corpus according to the implicit/explicit mismatch between different languages. Then we design a multi-task neural network model to incorporate our BiSynData to benefit implicit discourse relation recognition. Experimental results on both the English PDTB and Chinese CDTB data sets show that our proposed method achieves significant improvements over baselines using SynData.

A word alignment model based on multiobjective evolutionary algorithms

Word alignment is a key task in statistical machine translation (SMT). This paper presents a novel model for this task. In this model, word alignment is considered as a multiobjective optimization problem and solved based on the non-dominated sorting genetic algorithm II (NSGA-II), which is one of the best multiobjective evolutionary algorithms (MOEA). There are two advantages of the proposed model based on NSGA-II. First, it could be easily extended through incorporating new objective functions. Secondly, it does not need any hand-aligned word-level alignment data to determine the weight of each objective function. Experiments were carried out and the results show that the proposed model outperforms the IBM translation models significantly.

A model for ranking sentence pairs in parallel corpora

In this paper, the problem of ranking sentence pairs in parallel corpora was addressed for the first time. To solve this problem, a novel model was proposed. In this model, both syntax features and semantics features of sentence pairs are considered. Since most todays statistical machine translation models depend on word alignment, features related to word alignment information are also included. Two experiments were carried out and the results showed that the model had promising performance.

Constructing and validating word similarity datasets by integrating methods from psychology, brain science and computational linguistics

Human-scored word similarity gold-standard datasets are normally composed of word pairs with corresponding similarity scores. These datasets are popular resources for evaluating word similarity models which are the essential components for many natural language processing tasks. This paper proposes a novel multidisciplinary method for constructing and validating word similarity gold-standard datasets. The proposed method is different from the previous ones in that it introduces methods from three different disciplines, i.e., psychology, brain science and computational linguistics to validate the soundness of the constructed datasets. Specifically, to the best of our knowledge, this is the first time event-related potentials experiments are incorporated to validate the word similarity datasets. Using the proposed method, we finally constructed a Chinese gold-standard word similarity dataset with 260 word pairs and showed its soundness using the interdisciplinary validating methods. It should be noted that, although the paper only focused on constructing Chinese standard dataset, the proposed method is applicable to other languages.

Exploring Implicit Semantic Constraints for Bilingual Word Embeddings

Bilingual word embeddings (BWEs) have proven to be useful in many cross-lingual natural language processing tasks. Previous studies often require bilingual texts or dictionaries that are scarce resources. As a result, in these studies, the exploited explicit semantic information, such as monolingual word co-occurrences and cross-lingual semantic equivalences, is often insufficient for BWE learning, leading to the limitation of learned word representations. To overcome this problem, in this paper, we study how to exploit implicit semantic constraints for better BWEs. Concretely, we first discover implicit monolingual word-level semantic equivalences by pivoting their translations in the other language. Then, we perform BWE learning under various semantic constraints. Experimental results on machine translation and cross-lingual document classification demonstrate the effectiveness of our model.

Coping with Problems of Unicoded Traditional Mongolian

Traditional Mongolian Unicode Encoding has serious problems as several pairs of vowels with the same glyphs but different pronunciations are coded differently. We expose the severity of the problem by examples from our Mongolian corpus and propose two ways to alleviate the problem: first, developing a publicly available Mongolian input method that can help users to choose the correct encoding and second, a normalization method to solve the data sparseness problems caused by the proliferation of homographs. Experiments in search engines and statistical machine translation show that our methods are effective.

An Approach to N-Gram Language Model Evaluation in Phrase-Based Statistical Machine Translation

N-gram Language model plays an important role in statistical machine translation. Traditional methods adopt perplexity to evaluate language models, while this metric does not consider the characteristics of statistical machine translation. In this paper, we propose a novel method, namely bag-of-words decoding, to evaluate n-gram language models in phrase-based statistical machine translation. As compared with perplexity, our approach has more remarkable correlation with translation quality measured by BLEU. Experimental results on NIST data sets demonstrate the effectiveness of our method.