Barbara Di Eugenio

The kappa statistic: a second look

In recent years, the kappa coefficient of agreement has become the de facto standard for evaluating intercoder agreement for tagging tasks. In this squib, we highlight issues that affect and that the community has largely neglected. First, we discuss the assumptions underlying different computations of the expected agreement component of . Second, we discuss how prevalence and bias affect the measure.

Centering: A Parametric Theory and Its Instantiations

Centering theory is the best-known framework for theorizing about local coherence and salience; however, its claims are articulated in terms of notions which are only partially specified, such as utterance, realization, or ranking. A great deal of research has attempted to arrive at more detailed specifications of these parameters of the theory; as a result, the claims of centering can be instantiated in many different ways. We investigated in a systematic fashion the effect on the theorys claims of these different ways of setting the parameters. Doing this required, first of all, clarifying what the theorys claims are (one of our conclusions being that what has become known as Constraint 1 is actually a central claim of the theory). Secondly, we had to clearly identify these parametric aspects: For example, we argue that the notion of pronoun used in Rule 1 should be considered a parameter. Thirdly, we had to find appropriate methods for evaluating these claims. We found that while the theorys main claim about salience and pronominalization, Rule 1a preference for pronominalizing the backward-looking center (CB)is verified with most instantiations, Constraint 1a claim about (entity) coherence and CB uniquenessis much more instantiation-dependent: It is not verified if the parameters are instantiated according to very mainstream views (vanilla instantiation), it holds only if indirect realization is allowed, and is violated by between 20 and 25 of utterances in our corpus even with the most favorable instantiations. We also found a trade-off between Rule 1, on the one hand, and Constraint 1 and Rule 2, on the other: Setting the parameters to minimize the violations of local coherence leads to increased violations of salience, and vice versa. Our results suggest that entity coherencecontinuous reference to the same entitiesmust be supplemented at least by an account of relational coherence.

An effective Discourse Parser that uses Rich Linguistic Information

This paper presents a first-order logic learning approach to determine rhetorical relations between discourse segments. Beyond linguistic cues and lexical information, our approach exploits compositional semantics and segment discourse structure data. We report a statistically significant improvement in classifying relations over attribute-value learning paradigms such as Decision Trees, RIPPER and Naive Bayes. For discourse parsing, our modified shift-reduce parsing model that uses our relation classifier significantly outperforms a right-branching majority-class baseline.

The agreement process

In this paper, we investigate the empirical correlates of the agreement process. Informally, the agreement process is the dialog process by which collaborators achieve joint commitment on a joint action. We propose a specific instantiation of the agreement process, derived from our theoretical model, that integrates the IRMA framework for rational problem solving (Bratman, Israel & Pollack, 1988) with Clarks (1992, 1996) work on language as a collaborative activity; and from the characteristics of our task, a simple design problem (furnishing a two-room apartment) in which knowledge is equally distributed among agents, and needs to be shared. The main contribution of our paper is an empirical study of some of the components of the agreement process. We first discuss why we believe the findings from our corpus of computer-mediated dialogs are applicable to human?human collaborative dialogs in general. We then present our theoretical model, and apply it to make predictions about the components of the agreement process. We focus on how information is exchanged in order to arrive at a proposal, and on what constitutes a proposal and its acceptance/rejection. Our corpus study makes use of features of both the dialog and the domain reasoning situation, and led us to discover that the notion of commitment is more useful to model the agreement process than that of acceptance/rejection, as it more closely relates to the unfolding of negotiation.

Learning Features that Predict Cue Usage

Our goal is to identify the features that predict the occurrence and placement of discourse cues in tutorial explanations in order to aid in the automatic generation of explanations. Previous attempts to devise rules for text generation were based on intuition or small numbers of constructed examples. We apply a machine learning program, C4.5, to induce decision trees for cue occurrence and placement from a corpus of data coded for a variety of features previously thought to affect cue usage. Our experiments enable us to identify the features with most predictive power, and show that machine learning can be used to induce decision trees useful for text generation.

FLSA: Extending Latent Semantic Analysis with Features for Dialogue Act Classification

We discuss Feature Latent Semantic Analysis (FLSA), an extension to Latent Semantic Analysis (LSA). LSA is a statistical method that is ordinarily trained on words only; FLSA adds to LSA the richness of the many other linguistic features that a corpus may be labeled with. We applied FLSA to dialogue act classification with excellent results. We report results on three corpora: CallHome Spanish, MapTask, and our own corpus of tutoring dialogues.

Discourse Processing of Dialogues with Multiple Threads

In this paper we will present our ongoing work on a plan-based discourse processor developed in the context of the Enthusiast Spanish to English translation system as part of the JANUS multi-lingual speech-to-speech translation system. We will demonstrate that theories of discourse which postulate a strict tree structure of discourse on either the intentional or attentional level are not totally adequate for handling spontaneous dialogues. We will present our extension to this approach along with its implementation in our plan-based discourse processor. We will demonstrate that the implementation of our approach outperforms an implementation based on the strict tree structure approach.

Centering theory and the Italian pronominal system

In this paper, I give an account, in terms of centering theory [GJW86], of some phenomena of pronominalization in Italian, in particular the use of the null or the overt pronoun in subject position. After a general introduction to the Italian pronominal system, I will review centering, and then show how the original rules given in [GJW86] have to be extended or modified. Finally, I will show that centering does not account for two phenomena: first, the functional role of an utterance may override the predictions of centering; second, a null subject can be used to refer to a whole discourse segment. This later phenomenon should ideally be explained in the same terms that the other phenomena in volving null subject are.

UNDERSTANDING NATURAL LANGUAGE INSTRUCTIONS: THE CASE OF PURPOSE CLAUSES

This paper presents an analysis of purpose clauses in the context of instruction understanding. Such analysis shows that goals affect the interpretation and / or execution of actions, lends support to the proposal of using generation and enablement to model relations between actions, and sheds light on some inference processes necessary to interpret purpose clauses.

An Empirical Investigation of Proposals in Collaborative Dialogues

We describe a corpus-based investigation of proposals in dialogue. First, we describe our DRI compliant coding scheme and report our inter-coder reliability results. Next, we test several hypotheses about what constitutes a well-formed proposal.