Marta Recasens

Blanc: Implementing the rand index for coreference evaluation

This paper addresses the current state of coreference resolution evaluation, in which different measures (notably, MUC, B3, CEAF, and ACE-value) are applied in different studies. None of them is fully adequate, and their measures are not commensurate. We enumerate the desiderata for a coreference scoring measure, discuss the strong and weak points of the existing measures, and propose the BiLateral Assessment of Noun-Phrase Coreference, a variation of the Rand index created to suit the coreference task. The BiLateral Assessment of Noun-Phrase Coreference rewards both coreference and non-coreference links by averaging the F-scores of the two types, does not ignore singletons-the main problem with the MUC score-and does not inflate the score in their presence-a problem with the B3 and CEAF scores. In addition, its fine granularity is consistent over the whole range of scores and affords better discrimination between systems.

Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation

The definitions of two coreference scoring metrics- B3 and CEAF-are underspecified with respect to predicted, as opposed to key (or gold) mentions. Several variations have been proposed that manipulate either, or both, the key and predicted mentions in order to get a one-to-one mapping. On the other hand, the metric BLANC was, until recently, limited to scoring partitions of key mentions. In this paper, we (i) argue that mention manipulation for scoring predicted mentions is unnecessary, and potentially harmful as it could produce unintuitive results; (ii) illustrate the application of all these measures to scoring predicted mentions; (iii) make available an open-source, thoroughly-tested reference implementation of the main coreference evaluation measures; and (iv) rescore the results of the CoNLL-2011/2012 shared task systems with this implementation. This will help the community accurately measure and compare new end-to-end coreference resolution algorithms.

A Deeper Look into Features for Coreference Resolution

All automated coreference resolution systems consider a number of features, such as head noun, NP type, gender, or number. Although the particular features used is one of the key factors for determining performance, they have not received much attention, especially for languages other than English. This paper delves into a considerable number of pairwise comparison features for coreference, including old and novel features, with a special focus on the Spanish language. We consider the contribution of each of the features as well as the interaction between them. In addition, given the problem of class imbalance in coreference resolution, we analyze the effect of sample selection. From the experiments with TiMBL (Tilburg Memory-Based Learner) on the AnCora corpus, interesting conclusions are drawn from both linguistic and computational perspectives.

On paraphrase and coreference

By providing a better understanding of paraphrase and coreference in terms of similarities and differences in their linguistic nature, this article delimits what the focus of paraphrase extraction and coreference resolution tasks should be, and to what extent they can help each other. We argue for the relevance of this discussion to Natural Language Processing.

First-mention definites:More than exceptional cases

Traditional linguistic theories of definiteness have characterized the definite article in terms of uniqueness or familiarity, inclusiveness or identifiability. From this perspective, anaphoric uses of definite noun phrases (NPs) are seen as the paradigm case, while non-anaphoric or first-mention uses are treated as exceptions deserving no special attention. The main weaknesses of such approach are its tendency to be based on constructed examples and its focus on one single language, English. When natural data is taken into account, classical treatments of definites collapse.

Coreference resolution: an empirical study based on SemEval-2010 shared Task 1

This paper presents an empirical evaluation of coreference resolution that covers several interrelated dimensions. The main goal is to complete the comparative analysis from the SemEval-2010 task on Coreference Resolution in Multiple Languages. To do so, the study restricts the number of languages and systems involved, but extends and deepens the analysis of the system outputs, including a more qualitative discussion. The paper compares three automatic coreference resolution systems for three languages (English, Catalan and Spanish) in four evaluation settings, and using four evaluation measures. Given that our main goal is not to provide a comparison between resolution algorithms, these are merely used as tools to shed light on the different conditions under which coreference resolution is evaluated. Although the dimensions are strongly interdependent, making it very difficult to extract general principles, the study reveals a series of interesting issues in relation to coreference resolution: the portability of systems across languages, the influence of the type and quality of input annotations, and the behavior of the scoring measures.

A Chain-starting Classifier of Definite NPs in Spanish

Given the great amount of definite noun phrases that introduce an entity into the text for the first time, this paper presents a set of linguistic features that can be used to detect this type of definites in Spanish. The efficiency of the different features is tested by building a rule-based and a learning-based chain-starting classifier. Results suggest that the classifier, which achieves high precision at the cost of recall, can be incorporated as either a filter or an additional feature within a coreference resolution system to boost its performance.

Sense Anaphoric Pronouns: Am I One?

This paper focuses on identity-of-sense anaphoric relations, in which the sense is shared but not the referent. We are not restricted to the pronoun one, the focus of the small body of previous NLP work on this phenomenon, but look at a wider range of pronouns (that, some, another, etc.). We develop annotation guidelines, enrich a third of English OntoNotes with sense anaphora annotations, and shed light onto this phenomenon from a corpus-based perspective. We release the annotated data as part of the SAnaNotes corpus. We also use this corpus to develop a learning-based classifier to identify sense anaphoric uses, showing both the power and limitations of local features.

Annotated Corpora and Annotation Tools

In this chapter we review the currently available corpora to study anaphoric interpretation, and the tools that can be used to create new ones. A comprehensive survey of annotated corpora will be given, which ranges from the corpora and guidelines developed for the Message Understanding Conferences MUC-6 (1996) and MUC-7 (1998), which have been seminal to the field, to the resources that have been recently made available as part of the 2010 SemEval evaluation campaign. All fundamental design decisions regarding annotation formats and standards are described, and the relevant properties of the corpora are presented in a uniform and well-structured way. Moreover, three useful, widely used and freely available annotation tools (CALLISTO, MMAX2, and Palinka) will be described. They can be employed if own annotation work turns out to be indispensable.

Coreference is not always either/or: psycholinguistic evidence for near-identity

Coreference has been traditionally defined dichotomously as identity-of-reference or non-identity-of-reference. Here, we consider the existence of a near-identity category for referential relations that are neither coreferent nor non-coreferent. We present a three-task experiment on the interpretation of the identity relationship between the referents of noun phrases in English and in Catalan. The experiment collected the judgements and reaction times (RTs) of 70 English speakers and 34 Catalan speakers to investigate the reality of a near-identity category that undermines the traditional dichotomous approach to coreference. The results show that whereas some referential relations are classified as either identity or non-identity by the majority of participants, there is a third class of relations for which the judgements are split between identity and non-identity. This third class of relations also shows a longer RT. This evidence supports the conclusion that it does not suffice to distinguish between identity-of-reference and non-identity-of-reference, but that it is psychologically plausible to assume a middle-ground category of near-identity to include those referential relations on which participants disagree as to whether they are coreferent. In addition, the results allow to conclude that near-identity relations involve higher processing complexity. The fact that this is true for both English and Catalan points towards the cross-linguistic nature of near-identical referents.