Madeleine Bates

The BBN/HARC spoken language understanding system

The design and performance of a complete spoken language understanding system under development at BBN are described. The system, dubbed HARC (Hear And Respond to Continuous speech), successfully integrates state-of-the-art speech recognition and natural language understanding subsystems. The system has been tested extensively on a restricted airline travel information (ATIS) domain with a vocabulary of about 2000 words. HARC is implemented in portable, high-level software that runs in real time on todays workstations to support interactive online human-machine dialogs. No special-purpose hardware is required other than an A/D (analog-to-digital) converter to digitize the speech. The system works well for any native speaker of American English and does not require any enrollment data from the users. Results of formal DARPA tests in Feb. and Nov. 1992 are presented.<<ETX>>

Developing an evaluation methodology for spoken language systems

There has been a long-standing methodology for evaluating work in speech recognition (SR), but until recently no community-wide methodology existed for either natural language (NL) researchers or speech understanding (SU) researchers for evaluating the systems they developed.Recently considerable progress has been made by a number of groups involved in the DARPA Spoken Language Systems (SLS) program to agree on a methodology for comparative evaluation of SLS systems, and that methodology is being used in practice for the first time.This paper gives an overview of the process that was followed in creating a meaningful evaluation mechanism, describes the current mechanism, and presents some directions for future development.

A proposal for incremental dialogue evaluation

The SLS community has made progress recently toward evaluating SLS systems that deal with dialogue, but there is still considerable work that needs to be done in this area. Our goal is to develop incremental ways to evaluate dialogue processing, not just going from Class D1 (dialogue pairs) to Class D2 (dialogue triples), but measuring aspects of dialogue processing other than length. We present two suggestions; one for extending the common evaluation procedures for dialogues, and one for modifying the scoring metric.

The Delphi Natural Language Understanding System

This paper presents Delphi, the natural language component of the BBN Spoken Language System. Delphi is a domain-independent natural language question answering system that is solidly based on linguistic principles, yet which is also robust to ungrammatical input. It includes a domain-independent, broad-coverage grammar of English. Analysis components include an agenda-based best-first parser and a fallback component for partial understanding that works by fragment combination. Delphi has been formally evaluated in the ARPA Spoken Language programs ATIS (Airline Travel Information System) domain, and has performed well. Delphi has also been ported to a spoken language demonstration system in an Air Force Resource Management domain. We discuss results of the evaluation as well as the porting process.

BBN ATIS system progress report—June 1990

This paper reports recent progress on the development of the Delphi natural language component of the BBN spoken language system for the ATIS domain, focussing on the comparative evaluation performed by NIST in June, 1990.

CONTROLLED TRANSFORMATIONAL SENTENCE GENERATION

This paper describes a sentence generator that was built primarily to focus on syntactic form and syntactic relationships. Our main goal was to produce a tutorial system for the English language; the intended users of the system are people with language delaying handicaps such as deafness, and people learning English as a foreign language. For these populations, extensive exposure to standard English constructions (negatives, questions, relatlvization, etc.) and their interactions is necessary. • The purpose of the generator was to serve as a powerful resource for tutorial programs that need examples of particular constructions and/or related sentences to embed in exercises or examples for the student. The focus of the generator is thus not so much on what to express as on how to express it in acceptable English. This is quite different from the focus of most other language generation systems. Nonetheless, our system could be interfaced to a more goal-directed semantic component.

Critical challenges for natural language processing

Introduction Although natural language processing (NLP) has come very far in the last twenty years, the technology has not yet achieved a revolutionary impact on society. Is this because of some fundamental limitation that can never be overcome? Is it because there has not been enough time to refine and apply theoretical work that has already been done? We believe it is neither. We believe that several critical issues have never been adequately addressed in either theoretical or applied work, and that, because of a number of recent advances that we will discuss, the time is due for great leaps forward in the generality and utility of NLP systems. This paper focuses on roadblocks that seem surmountable within the next ten years. Rather than presenting new results, this paper identifies the problems that we believe must block widespread use of computational linguistics, and that can be solved within five to ten years. These are the problems that most need additional research and most deserve the talents and attention of Ph.D. students. We focus on the following areas, which will have maximum impact when combined in software systems: Knowledge acquisition from natural language (NL) texts of various kinds, from interactions with human beings, and from other sources. Language processing requires lexical, grammatical, semantic, and pragmatic knowledge. Current knowledge acquisition techniques are too slow and too difficult to use on a wide scale or on large problems. Knowledge bases should be many times the size of current ones. Interaction with multiple underlying systems to give NL systems the utility and flexibility demanded by people using them. Single application systems are limited in both usefulness and the language that is necessary to communicate with them. […]

Using spoken language to facilitate military transportation planning

The DARPA SLS Program is developing a technology that has been justified, at least in part, by its potential relevance to military applications. In an effort to demonstrate the relevance of SLS technology to real-world military applications, BBN has undertaken the task of providing a spoken language interface to DART, a system for military logistical transportation planning.We discuss the transportation planning process, describe the real-world DART system, identify parts of the system where spoken language can facilitate planning, and describe BBNs work towards porting the HARC SLS system to the DART domain.

Overview of the ARPA human language technology workshop

For five years, 1988-1992, the Defense Advanced Projects Agency sponsored a series of meetings called the DARPA Speech and Natural Language Workshops. These workshops provided a forum where researchers in speech and natural language, particularly as relating to the DARPA programs in spoken and written language understanding, could exchange information about recent research and technical progress.

BBN HARC and Delphi results on the ATIS benchmarks—February 1991

This paper presents the test results of running BBNs HARC spoken language system and DELPHI natural language understanding system on the ATIS benchmarks.We give a brief system overview, and review the major changes that have been made in Delphi since the last DARPA SLS workshop. We will briefly discuss the development and training process, and then present our test results and an analysis of their meaning.