Jette Viethen

Algorithms for Generating Referring Expressions: Do They Do What People Do?

The natural language generation literature provides many algorithms for the generation of referring expressions. In this paper, we explore the question of whether these algorithms actually produce the kinds of expressions that people produce. We compare the output of three existing algorithms against a data set consisting of human-generated referring expressions, and identify a number of significant differences between what people do and what these algorithms do. On the basis of these observations, we suggest some ways forward that attempt to address these differences.

Referring Expression Generation through Attribute-Based Heuristics

In this paper, we explore a corpus of human-produced referring expressions to see to what extent we can learn the referential behaviour the corpus represents. Despite a wide variation in the way subjects refer across a set of ten stimuli, we demonstrate that component elements of the referring expression generation process appear to generalise across participants to a significant degree. This leads us to propose an alternative way of thinking of referring expression generation, where each attribute in a description is provided by a separate heuristic.

Attribute-centric referring expression generation

In this chapter, we take the view that much of the existing work on the generation of referring expressions has focused on aspects of the problem that appear to be somewhat artificial when we look more closely at human-produced referring expressions. In particular, we argue that an over-emphasis on the extent to which each property in a description performs a discriminatory function has blinded us to alternative approaches to referring expression generation that might be better-placed to provide an explanation of the variety we find in human-produced referring expressions. On the basis of an analysis of a collection of such data, we propose an alternative view of the process of referring expression generation which we believe is more intuitively plausible, is a better match for the observed data, and opens the door to more sophisticated algorithms that are freed of the constraints adopted in the literature so far.

GRAPH: the costs of redundancy in referring expressions

We describe a graph-based generation system that participated in the TUNA attribute selection and realisation task of the REG 2008 Challenge. Using a stochastic cost function (with certain properties for free), and trying attributes from cheapest to more expensive, the system achieves overall .76 DICE and .54 MASI scores for attribute selection on the development set. For realisation, it turns out that in some cases higher attribute selection accuracy leads to larger differences between system-generated and human descriptions.

The GREC Main Subject Reference Generation Challenge 2009: Overview and Evaluation Results

The GREC-MSR Task at Generation Challenges 2009 required participating systems to select coreference chains to the main subject of short encyclopaedic texts collected from Wikipedia. Three teams submitted one system each, and we additionally created four baseline systems. Systems were tested automatically using existing intrinsic metrics. We also evaluated systems extrinsically by applying coreference resolution tools to the outputs and measuring the success of the tools. In addition, systems were tested in an intrinsic evaluation involving human judges. This report describes the GREC-MSR Task and the evaluation methods applied, gives brief descriptions of the participating systems, and presents the evaluation results.

The GREC Named Entity Generation Challenge 2009: Overview and Evaluation Results

The GREC-NEG Task at Generation Challenges 2009 required participating systems to select coreference chains for all people entities mentioned in short encyclopaedic texts about people collected from Wikipedia. Three teams submitted six systems in total, and we additionally created four baseline systems. Systems were tested automatically using a range of existing intrinsic metrics. We also evaluated systems extrinsically by applying coreference resolution tools to the outputs and measuring the success of the tools. In addition, systems were tested in an intrinsic evaluation involving human judges. This report describes the GREC-NEG Task and the evaluation methods applied, gives brief descriptions of the participating systems, and presents the evaluation results.

Referring in dialogue: alignment or construction?

Human speakers generally find it easy to refer to entities in such a way that their hearers can determine who or what is being talked about. In an attempt to model this behaviour, researchers in computational linguistics have explored the development of algorithms that operate in a deliberate manner, choosing attributes of an intended referent on the basis of their ability to distinguish that entity from its distractors. Psycholinguistic models, on the other hand, suggest that speakers align their referring expressions at several linguistic levels with those used previously in the discourse. This implies more subconscious reuse, and less deliberate choice, than is found in computational models of referring expression generation. Which of these is a more accurate characterisation of what people do? Do both models capture aspects of human referring behaviour? In this paper, we use a machine-learning approach to explore these questions. In our first study, we examine how underlying factors of the psycholinguistic and the computational models impact on the production of reference in dialogue. In our second study, we explore the psychological validity of another crucial aspect of some computational approaches to reference production: their serial dependency characteristic, whereby attributes are included in a referring expression based on which other attributes have already been chosen. The results of both studies suggest that the assumptions underpinning computational algorithms do not play a large role in peoples referring behaviour.

Realizing the Costs: Template-Based Surface Realisation in the GRAPH Approach to Referring Expression Generation

We describe a new realiser developed for the TUNA 2009 Challenge, and present its evaluation scores on the development set, showing a clear increase in performance compared to last years simple realiser.

Capturing Acceptable Variation in Distinguishing Descriptions

Almost all existing referring expression generation algorithms aim to find one best referring expression for a given intended referent. However, human-produced data demonstrates that, for any given entity, many perfectly acceptable referring expressions exist. At the same time, it is not the case that all logically possible descriptions are acceptable; so, if we remove the requirement to produce only one best solution, how do we avoid generating undesirable descriptions? Our aim in this paper is to sketch a framework that allows us to capture constraints on referring expression generation, so that the set of logically possible descriptions can be reduced to just those that are acceptable.