Sheryl R. Young

High level knowledge sources in usable speech recognition systems

The authors detail an integrated system which combines natural language processing with speech understanding in the context of a problem solving dialogue. The MINDS system uses a variety of pragmatic knowledge sources to dynamically generate expectations of what a user is likely to say.

Detecting misrecognitions and out-of-vocabulary words

This paper describes and evaluates a new technique for evaluating confidence in word strings produced by a speech recognition system. It detects misrecognized and out-of-vocabulary words in spontaneous spoken dialogs. The system uses multiple, diverse knowledge sources including acoustics, semantics, pragmatics and discourse to determine if a word string is misrecognized. When likely misrecognitions are detected, a series of tests distinguishes out-of-vocabulary words from other error sources. The work is part of a larger effort to automatically recognize and understand new words when spoken in a spontaneous spoken dialog. The newly developed acoustic confidence metrics output independent probabilites that a word is recognized correctly and a measure of how reliably we can tell if it is wrong. At p <.05, the acoustic methods detect 65% of the errors. The semantic/discourse module detects 98% of the errors that are semantically or contextually inappropriate, but cannot detect contextually consistent misrecognitions. Hence, we merged these two methods and ran them on a single test set to see whether the semantic/pragmatic/discourse component detected input not reliable rejected acoustically and to see how many of the semantically consistent errors could be detected with acoustic normalization methods.<<ETX>>

Flexible use of semantic constraints in speech recognition

A novel recursive transition network speech decoder designed for robust processing of spontaneous spoken input is described. Two levels of stochastic language models are used in the recognition search as well as the rule-based network constraints. The authors describe the basic decoder and system architecture and evaluate the system against a loosely coupled system on spontaneous spoken dialogues from the DARPA Air Travel Information System (ATIS) task.<<ETX>>

Use of dialogue, pragmatics and semantics to enhance speech recognition

Abstract Current, state-of-the-art speaker-independent continuous-speech recognizers are able to achieve word recognition rates in excess of 94 percent using lexicons of 1000 words or less and grammars or language models with perplexity 60 or less. Performance of these systems decreases rapidly as the perplexity of the grammar increases. As we allow users the flexibility to speak naturally, using constructions of their own choosing, perplexities increase more than an order of magnitude. Fortunately, knowledge of the domain and of communicative and problem solving behaviors can be used to dynamically decrease perplexity and allow more natural interaction given the current state of speech recognition technology. The perplexity reduction from knowledge results in speech performance equal to that demonstrated by speech recognizers using an equivalently low perplexity language model in the same or different domains. This paper addresses how knowledge of domain semantics, dialog, communication conventions and problem solving behavior are used to enhance automatic speech recognition and understanding. Included is a discussion of the systems basic principles and descriptions of the important knowledge sources and heuristics employed by the minds system. Prior perplexity reduction results are reviewed, demonstrating the systems ability to dynamically reduce perplexity and enhance recognition performance. This is followed by a brief analysis of some of the heuristics which do not have to be reimplemented across domains. Specifically addressed are why the heuristics are effective, and how much each can be expected to reduce entropy and average branching factor in any possible application domain.

Learning new words from spontaneous speech

The authors describe the design of a system to learn new words from spontaneous speech input, and present an initial experiment on detecting the new words to be learned. Learning a new word involves detecting an out-of-vocabulary word in the input, determining its meaning, and adding the word to the system lexicon and grammars. Such learning would enable later recognition, parsing, and interpretation of the new words.<<ETX>>

Speech understanding in open tasks

The Air Traffic Information Service task is currently used by DARPA as a common evaluation task for Spoken Language Systems. This task is an example of open type tasks. Subjects are given a task and allowed to interact spontaneously with the system by voice. There is no fixed lexicon or grammar, and subjects are likely to exceed those used by any given system. In order to evaluate system performance on such tasks, a common corpus of training data has been gathered and annotated. An independent test corpus was also created in a similar fashion. This paper explains the techniques used in our system and the performance results on the standard set of tests used to evaluate systems.

The MINDS system: using context and dialog to enhance speech

Abstract : Contextual knowledge has traditionally been used in multi-sentential textual understanding systems. In contrast, this paper describes a new approach toward using contextual, dialog-based knowledge for speech recognition. To demonstrate this approach, we have built MINDS, a system which uses contextual knowledge to predictively generate expectations about the conceptual content that may be expressed in a system users next utterance. These expectations are expanded to constrain the possible words which may be matched from an incoming speech signal. To prevent system rigidity and allow for diverse user behavior, the system creates layered predictions which range from very specific to very general. Each time new information becomes available from the ongoing dialog, MINDS generates a different set of layered predictions for processing the next utterance. The predictions contain constraints derived from the contextual, dialog level knowledge sources and each prediction is translated into a grammar usable by our speech recognizer, SPHINX. Since speech recognizers use grammars to dictate legal word sequences and to constrain the recognition process, the dynamically generated grammars reduce the number of word candidates considered by the recognizer. The results demonstrate that speech recognition accuracy is greatly enhanced through the use of predictions.

Towards Habitable Systems: Use of World Knowledge to Dynamically Constrain Speech Recognition

Publisher Summary Current state-of-the-art speaker-independent continuous speech recognizers are able to achieve word recognition rates well above 90 percent with lexicons of 1000 words or less using grammars with perplexity 60 or less. Performance of these systems decreases rapidly as the perplexity of the grammar increases. As users are allowed more flexibility in interacting with recognition systems, the size of the lexicons and perplexity of the grammars increase greatly. Allowing spontaneous speech instead of read speech, compounds the problems even more. Other sources of knowledge may be available to help constrain the ever more complex search spaces in such systems. When recognition systems are used in performing problem solving tasks, predictable features of the users behavior can be used to aid recognition. A system (MINDS), which uses additional constraints based on dialog interactions is also described in the chapter. The constraints are applied in a manner that allows optimum performance when users behave predictably, and degrades gracefully when they do not. An evaluation is also presented of the systems performance to show the utility of the additional knowledge sources.