David S. Pallett

Multi-site data collection and evaluation in spoken language understanding

The Air Travel Information System (ATIS) domain serves as the common task for DARPA spoken language system research and development. The approaches and results possible in this rapidly growing area are structured by available corpora, annotations of that data, and evaluation methods. Coordination of this crucial infrastructure is the charter of the Multi-Site ATIS Data COllection Working group (MADCOW). We focus here on selection of training and test data, evaluation of language understanding, and the continuing search for evaluation methods that will correlate well with expected performance of the technology in applications.

An acoustic‐phonetic data base

DARPA has sponsored the design and collection of a large speech data base. Six hundred and thirty speakers read ten sentences each. Two sentences were constant for all speakers; the remaining eight sentences were selected from a set of 450 designed at MIT and 1890 selected at TI from text sources. The set of sentences is phonetically rich, balanced, and deep. Although all recordings were made in Dallas, we sampled as many varieties of American English as possible. Selection of volunteer speakers was based on their childhood locality to give a balanced representation of geographical origins. The subject population is adult; 70% male; young (63% in their twenties); well educated (78% with bachelors degree); and predominantly white (96%). Recordings were made in a noise‐reducing sound booth using a Sennheiser headset microphone and digitized at 20 kHz. A natural reading style was encouraged. The recordings are complete, and time‐registered phonetic transcriptions are being added to the 6300 speech files at ...

Benchmark tests for DARPA resource management database performance evaluations

A nominally 1000-word resource management database for continuous speech recognition was developed for use in the DARPA Speech Research Program. This database has now been used at several sites for benchmark tests, and the database is expected to be made available to a wider community in the near future. The author documents the structure of the benchmark tests, including the selection of test material and details of studies of scoring algorithms.<<ETX>>

DARPA ATIS test results June 1990

The first Spoken Language System tests to be conducted in the DARPA Air Travel Information System (ATIS) domain took place during the period June 15 - 20, 1989. This paper presents a brief description of the test protocol, comparator software used for scoring results at NIST, test material selection process, and preliminary tabulation of the scored results for seven SLS systems from five sites: BBN, CMU, MIT/LCS, SRI and Unisys. One system, designated cmu-spi(r) in this paper, made use of digitized speech as input (.wav files), and generated CAS-format answers. Other systems made use of SNOR transcriptions (.snr files) as input.

Benchmark test procedures for automatic speech recognition systems

Benchmark test procedures have been developed and implemented in the DARPA‐sponsored program in automatic recognition of continuous speech. These tests make use of a database of recorded speech that includes material for both speaker‐dependent and speaker‐independent technologies. In implementing these uniform tests, it is necessary to define a standard lexicon of units to be scored and a standard or reference orthographic representation for each sentence, and to select a scoring procedure and error taxonomy. Considerations taken in identifying the reference lexicon and representations will be outlined. For the DARPA program, a dynamic‐programming string alignment procedure was implemented, and errors were defined in terms of substitutions, insertions, and deletions. This scoring software is publicly available. Known limitations of this procedure will be discussed. [Work supported by DARPA.]

National Institute of Standards and Technology (NIST): formerly National Bureau of Standards

Our objectives in the DARPA Spoken Language Program are: (1) to provide a central role in speech database design, development and distribution; and (2) to design, coordinate implementation of, and analyze results of performance tests.