Georgiana Dinu

Don't count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors

Context-predicting models (more commonly known as embeddings or neural language models) are the new kids on the distributional semantics block. Despite the buzz surrounding these models, the literature is still lacking a systematic comparison of the predictive models with classic, count-vector-based distributional semantic approaches. In this paper, we perform such an extensive evaluation, on a wide range of lexical semantics tasks and across many parameter settings. The results, to our own surprise, show that the buzz is fully justified, as the context-predicting models obtain a thorough and resounding victory against their count-based counterparts.

Hubness and Pollution: Delving into Cross-Space Mapping for Zero-Shot Learning

Zero-shot methods in language, vision and other domains rely on a cross-space mapping function that projects vectors from the relevant feature space (e.g., visualfeature-based image representations) to a large semantic word space (induced in an unsupervised way from corpus data), where the entities of interest (e.g., objects images depict) are labeled with the words associated to the nearest neighbours of the mapped vectors. Zero-shot cross-space mapping methods hold great promise as a way to scale up annotation tasks well beyond the labels in the training data (e.g., recognizing objects that were never seen in training). However, the current performance of cross-space mapping functions is still quite low, so that the strategy is not yet usable in practical applications. In this paper, we explore some general properties, both theoretical and empirical, of the cross-space mapping function, and we build on them to propose better methods to estimate it. In this way, we attain large improvements over the state of the art, both in cross-linguistic (word translation) and cross-modal (image labeling) zero-shot experiments.

Ranking Paraphrases in Context

We present a vector space model that supports the computation of appropriate vector representations for words in context, and apply it to a paraphrase ranking task. An evaluation on the SemEval 2007 lexical substitution task data shows promising results: the model significantly outperforms a current state of the art model, and our treatment of context is effective.

Inference Rules and their Application to Recognizing Textual Entailment

In this paper, we explore ways of improving an inference rule collection and its application to the task of recognizing textual entailment. For this purpose, we start with an automatically acquired collection and we propose methods to refine it and obtain more rules using a hand-crafted lexical resource. Following this, we derive a dependency-based structure representation from texts, which aims to provide a proper base for the inference rule application. The evaluation of our approach on the recognizing textual entailment data shows promising results on precision and the error analysis suggests possible improvements.

How to make words with vectors: Phrase generation in distributional semantics

We introduce the problem of generation in distributional semantics: Given a distributional vector representing some meaning, how can we generate the phrase that best expresses that meaning? We motivate this novel challenge on theoretical and practical grounds and propose a simple data-driven approach to the estimation of generation functions. We test this in a monolingual scenario (paraphrase generation) as well as in a cross-lingual setting (translation by synthesizing adjectivenoun phrase vectors in English and generating the equivalent expressions in Italian).

Picking buttercups and eating butter cups: Spelling alternations, semantic relatedness, and their consequences for compound processing

Semantic transparency (ST) is a measure quantifying the strength of meaning association between a compound word (buttercup) and its constituents (butter, cup). Borrowing ideas from computational semantics, we characterize ST in terms of the degree to which a compound and its constituents tend to share the same contexts in everyday usage, and we collect separate measures for different orthographic realizations (solid vs. open) of the same compound. We can thus compare the effects of semantic association in cases in which direct semantic access is likely to take place (buttercup), vis-a-vis forms that encourage combinatorial procedures (butter cup). ST effects are investigated in an analysis of lexical decision latencies. The results indicate that distributionally based ST variables are most predictive of response times when extracted from contexts presenting the compounds as open forms, suggesting that compound processing involves a conceptual combination procedure focusing on the merger of the constituent meanings.

Improving the Lexical Function Composition Model with Pathwise Optimized Elastic-Net Regression

In this paper, we show that the lexical function model for composition of distributional semantic vectors can be improved by adopting a more advanced regression technique. We use the pathwise coordinate-descent optimized elastic-net regression method to estimate the composition parameters, and compare the resulting model with several recent alternative approaches in the task of composing simple intransitive sentences, adjective-noun phrases and determiner phrases. Experimental results demonstrate that the lexical function model estimated by elastic-net regression achieves better performance, and it provides good qualitative interpretability through sparsity constraints on model parameters.

Weakly Supervised Cross-Lingual Named Entity Recognition via Effective Annotation and Representation Projection.

The state-of-the-art named entity recognition (NER) systems are supervised machine learning models that require large amounts of manually annotated data to achieve high accuracy. However, annotating NER data by human is expensive and time-consuming, and can be quite difficult for a new language. In this paper, we present two weakly supervised approaches for cross-lingual NER with no human annotation in a target language. The first approach is to create automatically labeled NER data for a target language via annotation projection on comparable corpora, where we develop a heuristic scheme that effectively selects good-quality projection-labeled data from noisy data. The second approach is to project distributed representations of words (word embeddings) from a target language to a source language, so that the source-language NER system can be applied to the target language without re-training. We also design two co-decoding schemes that effectively combine the outputs of the two projection-based approaches. We evaluate the performance of the proposed approaches on both in-house and open NER data for several target languages. The results show that the combined systems outperform three other weakly supervised approaches on the CoNLL data.

Word meaning in context: A probabilistic model and its application to Question Answering

The need for assessing similarity in meaning is central to most language technology applications. Distributional methods are robust, unsupervised methods which achieve high performance on this task. These methods measure similarity of word types solely based on patterns of word occurrences in large corpora, following the intuition that similar words occur in similar contexts. As most Natural Language Processing (NLP) applications deal with disambiguated words, words occurring in context, rather than word types, the question of adapting distributional methods to compute sense-specific or context-sensitive similarities has gained increasing attention in recent work. This thesis focuses on the development and applications of distributional methods for context-sensitive similarity. The contribution made is twofold: the main part of the thesis proposes and tests a new framework for computing similarity in context, while the second part investigates the application of distributional paraphrasing to the task of question answering. Die Notwendigkeit der Beurteilung von Bedeutungsahnlichkeit spielt fur die meisten sprachtechnologische Anwendungen eine wesentliche Rolle. Distributionelle Verfahren sind solide, unbeaufsichtigte Verfahren, die fur diese Aufgabe sehr effektiv sind. Diese Verfahren messen die Ahnlichkeit von Wortarten lediglich auf Basis von Mustern, nach denen die Worter in grosen Korpora vorkommen, indem sie der Erkenntnis folgen, dass ahnliche Worter in ahnlichen Kontexten auftreten. Da die meisten Anwendungen im Natural Language Processing (NLP) mit eindeutigen Wortern arbeiten, also eher Wortern, die im Kontext vorkommen, als Wortarten, hat die Frage, ob distributionelle Verfahren angepasst werden sollten, um bedeutungsspezifische oder kontextabhangige Ahnlichkeiten zu berechnen, in neueren Arbeiten zunehmend an Bedeutung gewonnen. Diese Dissertation konzentriert sich auf die Entwicklung und Anwendungen von distributionellen Verfahren fur kontextabhangige Ahnlichkeit und liefert einen doppelten Beitrag: Den Hauptteil der Arbeit bildet die Prasentation und Erprobung eines neuen framework fur die Berechnung von Ahnlichkeit im Kontext. Im zweiten Teil der Arbeit wird die Anwendung des distributional paraphrasing auf die Aufgabe der Fragenbeantwortung untersucht.

Inference Rules for Recognizing Textual Entailment

In this paper, we explore the application of inference rules for recognizing textual entailment (RTE). We start with an automatically acquired collection and then propose methods to refine it and obtain more rules using a hand-crafted lexical resource. Following this, we derive a dependency-based representation from texts, which aims to provide a proper base for the inference rule application. The evaluation of our approach on the RTE data shows promising results on precision and the error analysis suggests future improvements.