Deborah A. Dahl

Expanding the scope of the ATIS task: the ATIS-3 corpus

The Air Travel Information System (ATIS) domain serves as the common evaluation task for ARPA spoken language system developers. To support this task, the Multi-Site ATIS Data COllection Working group (MADCOW) coordinates data collection activities. This paper describes recent MADCOW activities. In particular, this paper describes the migration of the ATIS task to a richer relational database and development corpus (ATIS-3) and describes the ATIS-3 corpus. The expanded database, which includes information on 46 US and Canadian cities and 23,457 flights, was released in the fall of 1992, and data collection for the ATIS-3 corpus began shortly thereafter. The ATIS-3 corpus now consists of a total of 8297 released training utterances and 3211 utterances reserved for testing, collected at BBN, CMU, MIT, NIST and SRI. 2906 of the training utterances have been annotated with the correct information from the database. This paper describes the ATIS-3 corpus in detail, including breakdowns of data by type (e.g. context-independent, context-dependent, and unevaluable)and variations in the data collected at different sites. This paper also includes a description of the ATIS-3 database. Finally, we discuss future data collection and evaluation plans.

Recovering implicit information

This paper describes the SDC PUNDIT, (Prolog UNDerstands Integrated Text), system for processing natural language messages. PUNDIT, written in Prolog, is a highly modular system consisting of distinct syntactic, semantic and pragmatics components. Each component draws on one or more sets of data, including a lexicon, a broad-coverage grammar of English, semantic verb decompositions, rules mapping between syntactic and semantic constituents, and a domain model.This paper discusses the communication between the syntactic, semantic and pragmatic modules that is necessary for making implicit linguistic information explicit. The key is letting syntax and semantics recognize missing linguistic entities as implicit entities, so that they can be labelled as such, and reference resolution can be directed to find specific referents for the entities. In this way the task of making implicit linguistic information explicit becomes a subset of the tasks performed by reference resolution. The success of this approach is dependent on marking missing syntactic constituents as elided and missing semantic roles as ESSENTIAL so that reference resolution can know when to look for referents.

NOMINALIZATIONS IN PUNDIT

This paper describes the treatment of nominalizations in the PUNDIT text processing system. A single semantic definition is used for both nominalizations and the verbs to which they are related, with the same semantic roles, decompositions, and selectional restrictions on the semantic roles. However, because syntactically nominalizations are noun phrases, the processing which produces the semantic representation is different in several respects from that used for clauses. (1) The rules relating the syntactic positions of the constituents to the roles that they can fill are different. (2) The fact that nominalizations are untensed while clauses normally are tensed means that an alternative treatment of time is required for nominalizations. (3) Because none of the arguments of a nominalization is syntactically obligatory, some differences in the control of the filling of roles are required, in particular, roles can be filled as part of reference resolution for the nominalization. The differences in processing are captured by allowing the semantic interpreter to operate in two different modes, one for clauses, and one for nominalizations. Because many nominalizations are noun-noun compounds, this approach also addresses this problem, by suggesting a way of dealing with one relatively tractable subset of noun-noun compounds.

Beyond class A: a proposal for automatic evaluation of discourse

The DARPA Spoken Language community has just completed the first trial evaluation of spontaneous query/response pairs in the Air Travel (ATIS) domain.1 Our goal has been to find a methodology for evaluating correct responses to user queries. To this end, we agreed, for the first trial evaluation, to constrain the problem in several ways:Database Application: Constrain the application to a database query application, to ease the burden of a) constructing the back-end, and b) determining correct responses;

The PUNDIT natural-language processing system

The authors describe the PUNDIT (Prolog Understanding of Integrated Text) text-understanding system, which is designed to analyze and construct representations of paragraph-length text. PUNDIT is implemented in Quintus Prolog, and consists of distinct lexical, syntactic, semantic, and pragmatic components. Each component draws on one or more sets of data, including a lexicon, a broad-coverage grammar of English, semantic verb decompositions, rules mapping between syntactic and semantic constituents, and a domain model. Modularity, careful separation of declarative and procedural information, and separation of domain-specific and domain-independent information all contribute to a system which is flexible, extensible and portable. Versions of PUNDIT are now running in five domains, including four military and one medical.<<ETX>>

SENTENCE FRAGMENTS REGULAR STRUCTURES

This paper describes an analysis of telegraphic fragments as regular structures (not errors) handled by minimal extensions to a system designed for processing the standard language. The modular approach which has been implemented in the Unisys natural language processing system PUNDIT is based on a division of labor in which syntax regulates the occurrence and distribution of elided elements, and semantics and pragmatics use the systems standard mechanisms to interpret them.

Management and evaluation of interactive dialog in the air travel domain

This paper presents the Unisys Spoken Language System, as applied to the Air Travel Planning (ATIS) domain. This domain provides a rich source of interactive dialog, and has been chosen as a common application task for the development and evaluation of spoken language understanding systems. The Unisys approach to developing a spoken language system combines SUMMIT (the MIT speech recognition system [6]), PUNDIT (the Unisys language understanding system [3]) and an Ingres database of air travel information for eleven cities and nine airports (the ATIS database). Access to the database is mediated via a general knowledge-base/database interface (the Intelligent Database Server [4]). To date, we have concentrated on the language understanding and database interface components.

Training and evaluation of a spoken language understanding system

This paper describes our results on a spoken language application for finding directions. The spoken language system consists of the MIT SUMMIT speech recognition system ([20]) loosely coupled to the UNISYS PUNDIT language understanding system ([9]) with SUMMIT providing the top N candidates (based on acoustic score) to the PUNDIT system. The direction finding capability is provided by an expert system which is also part of the MIT VOYAGER system [18]).

Answers and questions: processing messages and queries

This paper describes issues in adapting the PUNDIT system, designed originally for message processing, to a query-answering system for the VOYAGER application. The resulting system, whose architecture and capabilities are described here, represents a first step towards our goal of demonstrating spoken language understanding in an interactive problem-solving context.

Recent improvements and benchmark results for the Paramax ATIS system

This paper describes three relatively domain-independent capabilities recently added to the Paramax spoken language understanding system: non-monotonic reasoning, implicit reference resolution, and database query paraphrase. In addition, we discuss the results of the February 1992 ATIS benchmark tests. We describe a variation on the standard evaluation metric which provides a more tightly controlled measure of progress. Finally, we briefly describe an experiment which we have done in extending the n-best speech/language integration architecture to improving OCR accuracy.