Stefan Thater

Translating Video Content to Natural Language Descriptions

Humans use rich natural language to describe and communicate visual perceptions. In order to provide natural language descriptions for visual content, this paper combines two important ingredients. First, we generate a rich semantic representation of the visual content including e.g. object and activity labels. To predict the semantic representation we learn a CRF to model the relationships between different components of the visual input. And second, we propose to formulate the generation of natural language as a machine translation problem using the semantic representation as source language and the generated sentences as target language. For this we exploit the power of a parallel corpus of videos and textual descriptions and adapt statistical machine translation to translate between our two languages. We evaluate our video descriptions on the TACoS dataset, which contains video snippets aligned with sentence descriptions. Using automatic evaluation and human judgments we show significant improvements over several baseline approaches, motivated by prior work. Our translation approach also shows improvements over related work on an image description task.

Assessing the impact of frame semantics on textual entailment

In this article, we underpin the intuition that frame semantic information is a useful resource for modelling textual entailment. To this end, we provide a manual frame semantic annotation for the test set used in the second recognizing textual entailment (RTE) challenge – the FrameNet-annotated textual entailment (FATE) corpus – and discuss experiments we conducted on this basis. In particular, our experiments show that the frame semantic lexicon provided by the Berkeley FrameNet project provides surprisingly good coverage for the task at hand. We identify issues of automatic semantic analysis components, as well as insufficient modelling of the information provided by frame semantic analysis as reasons for ambivalent results of current systems based on frame semantics.

A relational syntax-semantics interface based on dependency grammar

We propose a syntax-semantics interface that realises the mapping between syntax and semantics as a relation and does not make functionality assumptions in either direction. This interface is stated in terms of Extensible Dependency Grammar (XDG), a grammar formalism we newly specify. XDGs constraint-based parser supports the concurrent flow of information between any two levels of linguistic representation, even when only partial analyses are available. This generalises the concept of underspecification.

A Semantic Approach To Textual Entailment: System Evaluation and Task Analysis

This paper discusses our contribution to the third RTE Challenge -- the SALSA RTE system. It builds on an earlier system based on a relatively deep linguistic analysis, which we complement with a shallow component based on word overlap. We evaluate their (combined) performance on various data sets. However, earlier observations that the combination of features improves the overall accuracy could be replicated only partly.

Bridging the gap between underspecification formalisms: hole semantics as dominance constraints

We define a back-and-forth translation between Hole Semantics and dominance constraints, two formalisms used in underspecified semantics. There are fundamental differences between the two, but we show that they disappear on practically useful descriptions. Our encoding bridges a gap between two underspecification formalisms, and speeds up the processing of Hole Semantics.

Ranking Paraphrases in Context

We present a vector space model that supports the computation of appropriate vector representations for words in context, and apply it to a paraphrase ranking task. An evaluation on the SemEval 2007 lexical substitution task data shows promising results: the model significantly outperforms a current state of the art model, and our treatment of context is effective.

Minimal Recursion Semantics as Dominance Constraints: Translation, Evaluation, and Analysis

We show that a practical translation of MRS descriptions into normal dominance constraints is feasible. We start from a recent theoretical translation and verify its assumptions on the outputs of the English Resource Grammar (ERG) on the Redwoods corpus. The main assumption of the translation---that all relevant underspecified descriptions are nets---is validated for a large majority of cases; all non-nets computed by the ERG seem to be systematically incomplete.

Bridging the Gap Between Underspecification Formalisms: Minimal Recursion Semantics as Dominance Constraints

Minimal Recursion Semantics (MRS) is the standard formalism used in large-scale HPSG grammars to model underspecified semantics. We present the first provably efficient algorithm to enumerate the readings of MRS structures, by translating them into normal dominance constraints.

What Substitutes Tell Us - Analysis of an "All-Words" Lexical Substitution Corpus

We present the first large-scale English “allwords lexical substitution” corpus. The size of the corpus provides a rich resource for investigations into word meaning. We investigate the nature of lexical substitute sets, comparing them to WordNet synsets. We find them to be consistent with, but more fine-grained than, synsets. We also identify significant differences to results for paraphrase ranking in context reported for the SEMEVAL lexical substitution data. This highlights the influence of corpus construction approaches on evaluation results.

Efficient Solving and Exploration of Scope Ambiguities

We present the currently most efficient solver for scope underspecification; it also converts between different underspecification formalisms and counts readings. Our tool makes the practical use of large-scale grammars with (underspecified) semantic output more feasible, and can be used in grammar debugging.