Lynette Hirschman

Multi-site data collection and evaluation in spoken language understanding

The Air Travel Information System (ATIS) domain serves as the common task for DARPA spoken language system research and development. The approaches and results possible in this rapidly growing area are structured by available corpora, annotations of that data, and evaluation methods. Coordination of this crucial infrastructure is the charter of the Multi-Site ATIS Data COllection Working group (MADCOW). We focus here on selection of training and test data, evaluation of language understanding, and the continuing search for evaluation methods that will correlate well with expected performance of the technology in applications.

Multi-site data collection for a spoken language corpus

This paper describes a recently collected spoken language corpus for the ATIS (Air Travel Information System) domain. This data collection effort has been co-ordinated by MADCOW (Multi-site ATIS Data COllection Working group). We summarize the motivation for this effort, the goals, the implementation of a multi-site data collection paradigm, and the accomplishments of MADCOW in monitoring the collection and distribution of 12,000 utterances of spontaneous speech from five sites for use in a multi-site common evaluation of speech, natural language and spoken language.

Grammatically-based automatic word class formation

Abstract Most previous attempts at producing word classes (thesauri) by statistical analysis have used very limited distributional information such as word co-occurrence in a document or a sentence. This paper describes an automatic procedure which uses the syntactic relations as the basis for grouping words into classes. It forms classes by grouping together nouns that occur as subject (or object) of the same verbs, and similarly by grouping together verbs occurring with the same subject or object. The program was applied to a small corpus of sentences in a subfield of pharmacology. This procedure yielded the word classes for the subfield, in good agreement with the word classes recognized by pharmacologists. The word classes can be used to describe the informational patterns that occur in texts of the subfield, to disambiguate parses of a sentence, and perhaps to improve the performance of current information retrieval systems.

Recovering implicit information

This paper describes the SDC PUNDIT, (Prolog UNDerstands Integrated Text), system for processing natural language messages. PUNDIT, written in Prolog, is a highly modular system consisting of distinct syntactic, semantic and pragmatics components. Each component draws on one or more sets of data, including a lexicon, a broad-coverage grammar of English, semantic verb decompositions, rules mapping between syntactic and semantic constituents, and a domain model.This paper discusses the communication between the syntactic, semantic and pragmatic modules that is necessary for making implicit linguistic information explicit. The key is letting syntax and semantics recognize missing linguistic entities as implicit entities, so that they can be labelled as such, and reference resolution can be directed to find specific referents for the entities. In this way the task of making implicit linguistic information explicit becomes a subset of the tasks performed by reference resolution. The success of this approach is dependent on marking missing syntactic constituents as elided and missing semantic roles as ESSENTIAL so that reference resolution can know when to look for referents.

Experiments in evaluating interactive spoken language systems

As the DARPA spoken language community moves towards developing useful systems for interactive problem solving, we must explore alternative evaluation procedures that measure whether these systems aid people in solving problems within the task domain. In this paper, we describe several experiments exploring new evaluation procedures. To look at end-to-end evaluation, we modified our data collection procedure slightly in order to experiment with several objective task completion measures. We found that the task completion time is well correlated with the number of queries used. We also explored log file evaluation, where evaluators were asked to judge the clarity of the query and the correctness of the response based on examination of the log file. Our results show that seven evaluators were unanimous on more than 80% of the queries, and that at least 6 out of 7 evaluators agreed over 90% of the time. Finally, we applied these new procedures to compare two systems, one system requiring a complete parse and the other using the more flexible robust parsing mechanism. We found that these metrics could distinguish between these systems: there were significant differences in ability to complete the task, number of queries required to complete the task, and score (as computed through a log file evaluation) between the robust and the non-robust modes.

Beyond class A: a proposal for automatic evaluation of discourse

The DARPA Spoken Language community has just completed the first trial evaluation of spontaneous query/response pairs in the Air Travel (ATIS) domain.1 Our goal has been to find a methodology for evaluating correct responses to user queries. To this end, we agreed, for the first trial evaluation, to constrain the problem in several ways:Database Application: Constrain the application to a database query application, to ease the burden of a) constructing the back-end, and b) determining correct responses;

The PUNDIT natural-language processing system

The authors describe the PUNDIT (Prolog Understanding of Integrated Text) text-understanding system, which is designed to analyze and construct representations of paragraph-length text. PUNDIT is implemented in Quintus Prolog, and consists of distinct lexical, syntactic, semantic, and pragmatic components. Each component draws on one or more sets of data, including a lexicon, a broad-coverage grammar of English, semantic verb decompositions, rules mapping between syntactic and semantic constituents, and a domain model. Modularity, careful separation of declarative and procedural information, and separation of domain-specific and domain-independent information all contribute to a system which is flexible, extensible and portable. Versions of PUNDIT are now running in five domains, including four military and one medical.<<ETX>>

IMPROVED PORTABILITY AND PARSING THROUGH INTERACTIVE ACQUISITION OF SEMANTIC INFORMATION

This paper presents SPQR (Selectional Pattern Queries and Responses), a module of the PUNDIT text-processing system designed to facilitate the acquisition of domain-specific semantic information, and to improve the accuracy and efficiency of the parser. SPQR operates by interactively and incrementally collecting information about the semantic acceptability of certain lexical co-occurrence patterns (e.g., subject-verb-object) found in partially constructed parses. The module has proved to be a valuable tool for porting PUNDIT to new domains and acquiring essential semantic information about the domains. Preliminary results also indicate that SPQR causes a threefold reduction in the number of parses found, and about a 40% reduction in total parsing time.

Subject-based evaluation measures for interactive spoken language systems

The DARPA Spoken Language effort has profited greatly from its emphasis on tasks and common evaluation metrics. Common, standardized evaluation procedures have helped the community to focus research effort, to measure progress, and to encourage communication among participating sites. The task and the evaluation metrics, however, must be consistent with the goals of the Spoken Language program, namely interactive problem solving. Our evaluation methods have evolved with the technology, moving from evaluation of read speech from a fixed corpus through evaluation of isolated canned sentences to evaluation of spontaneous speech in context in a canned corpus. A key component missed in current evaluations is the role of subject interaction with the system. Because of the great variability across subjects, however, it is necessary to use either a large number of subjects or a within-subject design. This paper proposes a within-subject design comparing the results of a software-sharing exercise carried out jointly by MIT and SRI.

Question answering from natural language medical data bases

This paper describes a system for automatically answering questions about a collection of natural language medical records. The particular records used for an initial experiment were a set of 206 radiology reports. The implementation involves two major steps: manual determination of a suitable tabular structure (information format) for representing the information contained in the medical records, and automatic conversion of the natural language input (for either record or question) into a form corresponding to the data base. For the medical records the conversion into a data base is done by first performing a syntactic and transformational analysis of the sentences, followed by application of formatting transformations. The question-answering procedure has analogous initial steps but undergoes additional steps of processing to translate the question into a retrieval operation on the data base. Samples of the data base and of the question-answering procedure are shown.