Stan Szpakowicz

Beyond accuracy, f-score and ROC: a family of discriminant measures for performance evaluation

Different evaluation measures assess different characteristics of machine learning algorithms. The empirical evaluation of algorithms and classifiers is a matter of on-going debate among researchers. Most measures in use today focus on a classifiers ability to identify classes correctly. We note other useful properties, such as failure avoidance or class discrimination, and we suggest measures to evaluate such properties. These measures – Youdens index, likelihood, Discriminant power – are used in medical diagnosis. We show that they are interrelated, and we apply them to a case study from the field of electronic negotiations. We also list other learning problems which may benefit from the application of these measures.

Identifying expressions of emotion in text

Finding emotions in text is an area of research with wide-ranging applications. We describe an emotion annotation task of identifying emotion category, emotion intensity and the words/phrases that indicate emotion in text. We introduce the annotation scheme and present results of an annotation agreement study on a corpus of blog posts. The average inter-annotator agreement on labeling a sentence as emotion or non-emotion was 0.76. The agreement on emotion categories was in the range 0.6 to 0.79; for emotion indicators, it was 0.66. Preliminary results of emotion classification experiments show the accuracy of 73.89%, significantly above the baseline.

SemEval-2007 Task 04: Classification of Semantic Relations between Nominals

The NLP community has shown a renewed interest in deeper semantic analyses, among them automatic recognition of relations between pairs of words in a text. We present an evaluation task designed to provide a framework for comparing different approaches to classifying semantic relations between nominals in a sentence. This is part of SemEval, the 4th edition of the semantic evaluation event previously known as SensEval. We define the task, describe the training/test data and their creation, list the participating systems and discuss their results. There were 14 teams who submitted 15 systems.

Negoplan: an expert system shell for negotiation support

The authors address a complex, two-party negotiation problem containing the following elements: (1) many negotiation issues that are elements of a negotiating partys position; (2) negotiation goals that can be reduced to unequivocal statements about the problem domain and that represent negotiation issues; (3) a fluid negotiating environment characterized by changing issues and relations between them; and (4) parties negotiating to achieve goals that may change. They describe in some detail the way they logically specify different aspects of negotiation. An application of Negoplan to a labor contract negotiation between the Canadian Paperworkers Union and CIP, Ltd. of Montreal is described.<<ETX>>

Semi-Automatic Recognition of Noun Modifier Relationships

Semantic relationships among words and phrases are often marked by explicit syntactic or lexical clues that help recognize such relationships in texts. Within complex nominals, however, few overt clues are available. Systems that analyze such nominals must compensate for the lack of surface clues with other information. One way is to load the system with lexical semantics for nouns or adjectives. This merely shifts the problem elsewhere: how do we define the lexical semantics and build large semantic lexicons? Another way is to find constructions similar to a given complex nominal, for which the relationships are already known. This is the way we chose, but it too has drawbacks. Similarity is not easily assessed, similar analyzed constructions may not exist, and if they do exist, their analysis may not be appropriate for the current nominal.We present a semi-automatic system that identifies semantic relationships in noun phrases without using precoded noun or adjective semantics. Instead, partial matching on previously analyzed noun phrases leads to a tentative interpretation of a new input. Processing can start without prior analyses, but the early stage requires user interaction. As more noun phrases are analyzed, the system learns to find better interpretations and reduces its reliance on the user. In experiments on English technical texts the system correctly identified 60--70% of relationships automatically.

SemEval-2010 Task 9: The Interpretation of Noun Compounds Using Paraphrasing Verbs and Prepositions

We present a brief overview of the main challenges in understanding the semantics of noun compounds and consider some known methods. We introduce a new task to be part of SemEval-2010: the interpretation of noun compounds using paraphrasing verbs and prepositions. The task is meant to provide a standard testbed for future research on noun compound semantics. It should also promote paraphrase-based approaches to the problem, which can benefit many NLP applications.

Classification of semantic relations between nominals

The NLP community has shown a renewed interest in deeper semantic analyses, among them automatic recognition of semantic relations in text. We present the development and evaluation of a semantic analysis task: automatic recognition of relations between pairs of nominals in a sentence. The task was part of SemEval-2007, the fourth edition of the semantic evaluation event previously known as SensEval. Apart from the observations we have made, the long-lasting effect of this task may be a framework for comparing approaches to the task. We introduce the problem of recognizing relations between nominals, and in particular the process of drafting and refining the definitions of the semantic relations. We show how we created the training and test data, list and briefly describe the 15 participating systems, discuss the results, and conclude with the lessons learned in the course of this exercise.

Representing the negotiation process with a rule-based formalism

The objective of this paper is to introduce a flexible approach to the structuring of negotiations. The process of negotiations with its intricacies is discussed, and drawbacks of quantitative methods are analyzed. The decomposition of the negotiation process into a certain hierarchical structure is presented. This structure is represented with ‘and/or’ trees used for knowledge representation in artificial intelligence. The definitions of flexibility and reactions to the opponents moves are introduced with the help of a rule-based formalism. The implications of these definitions for the analysis of the negotiation process are presented. The approach is illustrated with a set of hypothetical examples.

Semi-automatic acquisition of conceptual structure from technical texts

We present a system which processes technical text semi-automatically andincrementally builds a conceptual model of the domain. Starting from an initial general model, knowledge-based text understanding is turned into knowledge acquisition. Incompletely understood text fragments may contain new information which should be integrated into the model under the control of an operator. The text is assumed to describe the domain fully. Typical problems in this domain are assumed to be solvable by indicating activities which manipulate objects. Activities, objects and their properties enter relationships that form a conceptual network. To test our representation, we have created a large hierarchy of concepts for PowerHouse Quiz. The system relies in its operation on the text and the growing network; it includes a parser with broad syntactic coverage, and a matcher retrieving subnetworks relevant to the current text fragment. The frequency of the operators necessary interventions depends on the initial networks size which will be determined experimentally. We discuss the status of the system and outline further work.

The Design and Implementation of an Electronic Lexical Knowledge Base

Thesauri have always been a useful resource for natural language processing. WordNet, a kind of thesaurus, has proven invaluable in computational linguistics. We present the various applications of Rogets Thesaurus in this field and discuss the advantages of its structure. We evaluate the merits of the 1987 edition of Penguins Rogets Thesaurus of English Words and Phrases as an NLP resource: we design and implement an electronic lexical knowledge base with its material. An extensive qualitative and quantitative comparison of Rogets and WordNet has been performed, and the ontologies as well as the semantic relations of both thesauri contrasted. We discuss the design in Java of the lexical knowledge base, and its potential applications. We also propose a framework for measuring similarity between concepts and annotating Rogets semantic links with WordNet labels.