Malvina Nissim

The Meaning Factory: Formal Semantics for Recognizing Textual Entailment and Determining Semantic Similarity

Shared Task 1 of SemEval-2014 comprised two subtasks on the same dataset of sentence pairs: recognizing textual entailment and determining textual similarity. We used an existing system based on formal semantics and logical inference to participate in the first subtask, reaching an accuracy of 82%, ranking in the top 5 of more than twenty participating systems. For determining semantic similarity we took a supervised approach using a variety of features, the majority of which was produced by our system for recognizing textual entailment. In this subtask our system achieved a mean squared error of 0.322, the best of all participating systems.

SemEval-2007 Task 08: Metonymy Resolution at SemEval-2007

We provide an overview of the metonymy resolution shared task organised within SemEval-2007. We describe the problem, the data provided to participants, and the evaluation measures we used to assess performance. We also give an overview of the systems that have taken part in the task, and discuss possible directions for future work.

Data and models for metonymy resolution

We describe the first shared task for figurative language resolution, which was organised within SemEval-2007 and focused on metonymy. The paper motivates the linguistic principles of data sampling and annotation and shows the task’s feasibility via human agreement. The five participating systems mainly used supervised approaches exploiting a variety of features, of which grammatical relations proved to be the most useful. We compare the systems’ performance to automatic baselines as well as to a manually simulated approach based on selectional restriction violations, showing some limitations of this more traditional approach to metonymy recognition. The main problem supervised systems encountered is data sparseness, since metonymies in general tend to occur more rarely than literal uses. Also, within metonymies, the reading distribution is skewed towards a few frequent metonymy types. Future task developments should focus on addressing this issue.

Adding Semantics to Data-Driven Paraphrasing

We add an interpretable semantics to the paraphrase database (PPDB). To date, the relationship between phrase pairs in the database has been weakly defined as approximately equivalent. We show that these pairs represent a variety of relations, including directed entailment (little girl/girl) and exclusion (nobody/someone). We automatically assign semantic entailment relations to entries in PPDB using features derived from past work on discovering inference rules from text and semantic taxonomy induction. We demonstrate that our model assigns these relations with high accuracy. In a downstream RTE task, our labels rival relations from WordNet and improve the coverage of a proof-based RTE system by 17%.

Modeling the internal variability of multiword expressions through a pattern-based method

The issue of internal variability of multiword expressions (MWEs) is crucial towards their identification and extraction in running text. We present a corpus-supported and computational study on Italian MWEs, aimed at defining an automatic method for modeling internal variation, exploiting frequency and part-of-speech (POS) information. We do so by deriving an XML-encoded lexicon of MWEs based on a manually compiled dictionary, which is then projected onto a a large corpus. Since a search for fixed forms suffers from low recall, while an unconstrained flexible search for lemmas yields a loss in precision, we suggest a procedure aimed at maximizing precision in the identification of MWEs within a flexible search. Our method builds on the idea that internal variability can be modelled via the novel introduction of variation patterns, which work over POS patterns, and can be used as working tools for controlling precision. We also compare the performance of variation patterns to that of association measures, and explore the possibility of using variation patterns in MWE extraction in addition to identification. Finally, we suggest that corpus-derived, pattern-related information can be included in the original MWE lexicon by means of an enriched coding and the creation of an XML-based repository of patterns.

Senso Comune: A Collaborative Knowledge Resource for Italian

Senso Comune is an open knowledge base for the Italian language, available through a Web-based collaborative platform, whose construction is in progress. The resource integrates dictionary data coming from both users and legacy resources with an ontological backbone, which provides foundations for a formal characterization of lexical semantic structures (frames). A nucleus of basic Italian lemmas, which have been semantically analyzed and classified, is available for both online access and download. A restricted community of contributors is currently working on increasing the lexical coverage of the resource.

Automatic identification of semantic relations in Italian complex nominals

This paper addresses the problem of the identification of the semantic relations in Italian complex nominals (CNs) of the type N+P+N. We exploit the fact that the semantic relation, which is underspecified in most cases, is partially made explicit by the preposition. We develop an annotation framework around five different semantic relations, which we use to create a corpus of 1700 Italian CNs, obtaining an inter-annotator agreement of K=.695. Exploiting this data, for each preposition p we train a classifier to assign one of the five semantic relations to any CN of the type N+p+N, by using both string and supersense features. To obtain supersenses, we experiment with a sequential tagger as well as a plain lookup in MultiWordNet, and find that using information obtained from the former yields better results.

Semantic Aspects in Sentiment Analysis

Abstract Semantics plays an important role in the accurate analysis of the context of a sentiment expression. We analyze this role from two perspectives: the way semantics is encoded in sentiment resources, such as lexica, corpora, and ontologies, and the way it is used by automatic systems that perform sentiment analysis on social media data. Specifically, we review and discuss state-of-the-art methods and tools that rely on semantic models and resources, possibly enabling reasoning, so as to deal with some key challenges for sentiment analysis. Examples of such challenges are context dependency and finer-grained sentiment detection, which can translate into the assigning of sentiment values to aspects of objects or into the design and use of a wider spectrum of affective labels, or the use of current techniques in finer-grained semantic processing. In our dealing with semantics, lexical aspects must be combined with pragmatic and cognitive issues, and other dimensions involved in conveying sentiment, such as emotions.

An Analysis of Cross-Genre and In-Genre Performance for Author Profiling in Social Media

User profiling on social media data is normally done within a supervised setting. A typical feature of supervised models that are trained on data from a specific genre, is their limited portability to other genres. Cross-genre models were developed in the context of PAN 2016, where systems were trained on tweets, and tested on other non-tweet social media data. Did the model that achieved best results at this task got lucky or was it truly designed in a cross-genre manner, with features general enough to capture demographics beyond Twitter? We explore this question via a series of in-genre and cross-genre experiments on English and Spanish using the best performing system at PAN 2016, and discover that portability is successful to a certain extent, provided that the sub-genres involved are close enough. In such cases, it is also more beneficial to do cross-genre than in-genre modelling if the cross-genre setting can benefit from larger amounts of training data than those available in-genre.

Answer translation: an alternative approach to cross-lingual question answering

We approach cross-lingual question answering by using a mono-lingual QA system for the source language and by translating resulting answers into the target language. As far as we are aware, this is the first cross-lingual QA system in the history of CLEF that uses this method--almost without exception, cross-lingual QA systems use translation of the question or query terms instead. We demonstrate the feasibility of our alternative approach by using a mono-lingual QA system for English, and translating answers and finding appropriate documents in Italian and Dutch. For factoid and definition questions, we achieve overall accuracy scores ranging from 13% (EN→NL) to 17% (EN→IT) and lenient accuracy figures from 19% (EN→NL) to 25% (EN→IT). The advantage of this strategy to cross-lingual QA is that translation of answers is easier than translating questions--the disadvantage is that answers might be missing from the source corpus and additional effort is required for finding supporting documents of the target language.