Lewis M. Norton

Beyond class A: a proposal for automatic evaluation of discourse

The DARPA Spoken Language community has just completed the first trial evaluation of spontaneous query/response pairs in the Air Travel (ATIS) domain.1 Our goal has been to find a methodology for evaluating correct responses to user queries. To this end, we agreed, for the first trial evaluation, to constrain the problem in several ways:Database Application: Constrain the application to a database query application, to ease the burden of a) constructing the back-end, and b) determining correct responses;

Management and evaluation of interactive dialog in the air travel domain

This paper presents the Unisys Spoken Language System, as applied to the Air Travel Planning (ATIS) domain. This domain provides a rich source of interactive dialog, and has been chosen as a common application task for the development and evaluation of spoken language understanding systems. The Unisys approach to developing a spoken language system combines SUMMIT (the MIT speech recognition system [6]), PUNDIT (the Unisys language understanding system [3]) and an Ingres database of air travel information for eleven cities and nine airports (the ATIS database). Access to the database is mediated via a general knowledge-base/database interface (the Intelligent Database Server [4]). To date, we have concentrated on the language understanding and database interface components.

Training and evaluation of a spoken language understanding system

This paper describes our results on a spoken language application for finding directions. The spoken language system consists of the MIT SUMMIT speech recognition system ([20]) loosely coupled to the UNISYS PUNDIT language understanding system ([9]) with SUMMIT providing the top N candidates (based on acoustic score) to the PUNDIT system. The direction finding capability is provided by an expert system which is also part of the MIT VOYAGER system [18]).

Answers and questions: processing messages and queries

This paper describes issues in adapting the PUNDIT system, designed originally for message processing, to a query-answering system for the VOYAGER application. The resulting system, whose architecture and capabilities are described here, represents a first step towards our goal of demonstrating spoken language understanding in an interactive problem-solving context.

Recent improvements and benchmark results for the Paramax ATIS system

This paper describes three relatively domain-independent capabilities recently added to the Paramax spoken language understanding system: non-monotonic reasoning, implicit reference resolution, and database query paraphrase. In addition, we discuss the results of the February 1992 ATIS benchmark tests. We describe a variation on the standard evaluation metric which provides a more tightly controlled measure of progress. Finally, we briefly describe an experiment which we have done in extending the n-best speech/language integration architecture to improving OCR accuracy.

A portable approach to last resort parsing and interpretation

This paper describes an approach to robust processing which is domain-independent in its design, yet which can easily take advantage of domain-specific information. Robust processing is well-integrated into standard processing in this approach, requiring essentially only a single new BNF rule in the grammar. We describe the results of implementing this approach in two different domains.

Integrated text and image understanding for document understanding

Because of the complexity of documents and the variety of applications which must be supported, document understanding requires the integration of image understanding with text understanding. Our document understanding technology is implemented in a system called IDUS (Intelligent Document Understanding System), which creates the data for a text retrieval application and the automatic generation of hypertext links. This paper summarizes the areas of research during IDUS development where we have found the most benefit from the integration of image and text understanding.

Augmented role filling capabilities for semantic interpretation of spoken language

This paper describes recent work on the Unisys ATIS Spoken Language System, and reports benchmark results on natural language, spoken language, and speech recognition. We describe enhancements to the systems semantic processing for handling non-transparent argument structure and enhancements to the systems pragmatic processing of material in answers displayed to the user. We found that the systems score on the natural language benchmark test decreased from 48% to 36% without these enhancements. We also report results for three spoken language systems, Unisys natural language coupled with MIT-Summit speech recognition, Unisys natural language coupled with MIT-Lincoln Labs speech recognition and Unisys natural language coupled with BBN speech recognition. Speech recognition results are reported on the results of the Unisys natural language selecting a candidate from the MIT-Summit N-best (N=16).

Integrating Natural Language Understanding with Document Structure Analysis

Document understanding, the interpretation of a document from its image form, is a technology area which benefits greatly from the integration of natural language processing with image processing. We have developed a prototype of an Intelligent Document Understanding System (IDUS) which employs several technologies: image processing, optical character recognition, document structure analysis and text understanding in a cooperative fashion. This paper discusses those areas of research during development of IDUS where we have found the most benefit from the integration of natural language processing and image processing: document structure analysis, optical character recognition (OCR) correction, and text analysis. We also discuss two applications which are supported by IDUS: text retrieval and automatic generation of hypertext links

A Test Environment for Natural Language Understanding Systems

The Natural Language Understanding Engine Test Environment (ETE) is a GUI software tool that aids in the development and maintenance of large, modular, natural language understanding (NLU) systems. Natural language understanding systems are composed of modules (such as part-of-speech taggers, parsers and semantic analyzers) which are difficult to test individually because of the complexity of their output data structures. Not only are the output data structures of the internal modules complex, but also many thousands of test items (messages or sentences) are required to provide a reasonable sample of the linguistic structures of a single human language, even if the language is restricted to a particular domain. The ETE assists in the management and analysis of the thousands of complex data structures created during natural language processing of a large corpus using relational database technology in a network environment.