Massimo Poesio

Inter-coder agreement for computational linguistics

This article is a survey of methods for measuring agreement among corpus annotators. It exposes the mathematics and underlying assumptions of agreement coefficients, covering Krippendorffs alpha as well as Scotts pi and Cohens kappa; discusses the use of coefficients in several annotation tasks; and argues that weighted, alpha-like coefficients, traditionally less used than kappa-like measures in computational linguistics, may be more appropriate for many corpus annotation tasksbut that their use makes the interpretation of the value of the coefficient even harder.

The TRAINS Project: A Case Study in Defining a Conversational Planning Agent

The TRAINS project is an effort to build a conversationally proficient planning assistant. A key part of the project is the construction of the TRAINS system, which provides the research platform for a wide range of issues in natural language understanding, mixed-initiative planning systems, and representing and reasoning about time, actions and events. Four years have now passed since the beginning of the project. Each year we have produced a demonstration system that focused on a dialog that illustrates particular aspects of our research. The commitment to building complete integrated systems is a significant overhead on the research, but we feel it is essential to guarantee that the results constitute real progress in the field. This paper describes the goals of the project, and our experience with the effort so far. .pp This paper is to appear in the Journal of Experimental and Theoretical AI, 1995.

An empirically based system for processing definite descriptions

We present an implemented system for processing definite descriptions in arbitrary domains. The design of the system is based on the results of a corpus analysis previously reported, which highlighted the prevalence of discourse-new descriptions in newspaper corpora. The annotated corpus was used to extensively evaluate the proposed techniques for matching definite descriptions with their antecedents, discourse segmentation, recognizing discourse-new descriptions, and suggesting anchors for bridging descriptions.

Centering: A Parametric Theory and Its Instantiations

Centering theory is the best-known framework for theorizing about local coherence and salience; however, its claims are articulated in terms of notions which are only partially specified, such as utterance, realization, or ranking. A great deal of research has attempted to arrive at more detailed specifications of these parameters of the theory; as a result, the claims of centering can be instantiated in many different ways. We investigated in a systematic fashion the effect on the theorys claims of these different ways of setting the parameters. Doing this required, first of all, clarifying what the theorys claims are (one of our conclusions being that what has become known as Constraint 1 is actually a central claim of the theory). Secondly, we had to clearly identify these parametric aspects: For example, we argue that the notion of pronoun used in Rule 1 should be considered a parameter. Thirdly, we had to find appropriate methods for evaluating these claims. We found that while the theorys main claim about salience and pronominalization, Rule 1a preference for pronominalizing the backward-looking center (CB)is verified with most instantiations, Constraint 1a claim about (entity) coherence and CB uniquenessis much more instantiation-dependent: It is not verified if the parameters are instantiated according to very mainstream views (vanilla instantiation), it holds only if indirect realization is allowed, and is violated by between 20 and 25 of utterances in our corpus even with the most favorable instantiations. We also found a trade-off between Rule 1, on the one hand, and Constraint 1 and Rule 2, on the other: Setting the parameters to minimize the violations of local coherence leads to increased violations of salience, and vice versa. Our results suggest that entity coherencecontinuous reference to the same entitiesmust be supplemented at least by an account of relational coherence.

CONVERSATIONAL ACTIONS AND DISCOURSE SITUATIONS

We use the idea that actions performed in a conversation become part of the common ground as the basis for a model of context that reconciles in a general and systematic fashion the differences between the theories of discourse context used for reference resolution, intention recognition, and dialogue management. We start from the treatment of anaphoric accessibility developed in discourse representation theory (DRT), and we show first how to obtain a discourse model that, while preserving DRTs basic ideas about referential accessibility, includes information about the occurrence of speech acts and their relations. Next, we show how the different kinds of ‘structure’ that play a role in conversation—discourse segmentation, turn‐taking, and grounding—can be formulated in terms of information about speech acts, and use this same information as the basis for a model of the interpretation of fragmentary input.

Strudel: A Corpus‐Based Semantic Model Based on Properties and Types

Computational models of meaning trained on naturally occurring text successfully model human performance on tasks involving simple similarity measures, but they characterize meaning in terms of undifferentiated bags of words or topical dimensions. This has led some to question their psychological plausibility (Murphy, 2002;Schunn, 1999). We present here a fully automatic method for extracting a structured and comprehensive set of concept descriptions directly from an English part-of-speech-tagged corpus. Concepts are characterized by weighted properties, enriched with concept-property types that approximate classical relations such as hypernymy and function. Our model outperforms comparable algorithms in cognitive tasks pertaining not only to concept-internal structures (discovering properties of concepts, grouping properties by property type) but also to inter-concept relations (clustering into superordinates), suggesting the empirical validity of the property-based approach.

Learning to Resolve Bridging References

We use machine learning techniques to find the best combination of local focus and lexical distance features for identifying the anchor of mereological bridging references. We find that using first mention, utterance distance, and lexical distance computed using either Google or WordNet results in an accuracy significantly higher than obtained in previous experiments.

Resolving bridging references in unrestricted text

Our goal is to develop a system capable of treating the largest possible subset of definite descriptions in unrestricted written texts. A previous prototype resolved anaphoric uses of definite descriptions and identified some types of first-mention uses, achieving a recall of 56%. In this paper we present the latest version of our system, which handles some types of bridging references, uses WordNet as a source of lexical knowledge, and achieves a recall of 65%.

BART: A Modular Toolkit for Coreference Resolution

Developing a full coreference system able to run all the way from raw text to semantic interpretation is a considerable engineering effort, yet there is very limited availability of off-the shelf tools for researchers whose interests are not in coreference, or for researchers who want to concentrate on a specific aspect of the problem. We present BART, a highly modular toolkit for developing coreference applications. In the Johns Hopkins workshop on using lexical and encyclopedic knowledge for entity disambiguation, the toolkit was used to extend a reimplementation of the Soon et al. (2001) proposal with a variety of additional syntactic and knowledge-based features, and experiment with alternative resolution processes, preprocessing tools, and classifiers.

The Reliability of Anaphoric Annotation, Reconsidered: Taking Ambiguity into Account

We report the results of a study of the reliability of anaphoric annotation which (i) involved a substantial number of naive subjects, (ii) used Krippendorffs α instead of K to measure agreement, as recently proposed by Passonneau, and (iii) allowed annotators to mark anaphoric expressions as ambiguous.