Nikos Tsourakis

A Bidirectional Grammar-Based Medical Speech Translator

We describe a bidirectional version of the grammar-based MedSLT medical speech system. The system supports simple medical examination dialogues about throat pain between an English-speaking physician and a Spanish-speaking patient. The physicians side of the dialogue is assumed to consist mostly of WH-questions, and the patients of elliptical answers. The paper focusses on the grammar-based speech processing architecture, the ellipsis resolution mechanism, and the online help system.

An Open Web Platform for Rule-Based Speech-to-Sign Translation.

We present an open web platform for developing, compiling, and running rulebased speech to sign language translation applications. Speech recognition is performed using the Nuance Recognizer 10.2 toolkit, and signed output, including both manual and non-manual components, is rendered using the JASigning avatar system. The platform is designed to make the component technologies readily accessible to sign language experts who are not necessarily computer scientists. Translation grammars are written in a version of Synchronous Context-Free Grammar adapted to the peculiarities of sign language. All processing is carried out on a remote server, with content uploaded and accessed through a web interface. Initial experiences show that simple translation grammars can be implemented on a time-scale of a few hours to a few days and produce signed output readily comprehensible to Deaf informants. Overall, the platform drastically lowers the barrier to entry for researchers interested in building applications that generate high-quality signed language.

Lightweight Spoken Utterance Classification with CFG, tf-idf and Dynamic Programming

We describe a simple spoken utterance classification method suitable for data-sparse domains which can be approximately described by CFG grammars. The central idea is to perform robust matching of CFG rules against output from a large-vocabulary recogniser, using a dynamic programming method which optimises the tf-idf score of the matched grammar string. We present results of experiments carried out on a substantial CFG-based medical speech translator and the publicly available Spoken CALL Shared Task. Robust utterance classification using the tf-idf method strongly outperforms plain CFG-based recognition for both domains. When comparing with Naive Bayes classifiers trained on data sampled from the CFG grammars, the tf-idf/dynamic programming method is much better on the complex speech translation domain, but worse on the simple Spoken CALL Shared Task domain.

Discriminative learning using linguistic features to rescore n-best speech hypotheses

We describe how we were able to improve the accuracy of a medium-vocabulary spoken dialog system by rescoring the list of n-best recognition hypotheses using a combination of acoustic, syntactic, semantic and discourse information. The non-acoustic features are extracted from different intermediate processing results produced by the natural language processing module, and automatically filtered. We apply discriminative support vector learning designed for re-ranking, using both word error rate and semantic error rate as ranking target value, and evaluating using five-fold cross-validation; to show robustness of our method, confidence intervals for word and semantic error rates are computed via bootstrap sampling. The reduction in semantic error rate, from 19% to 11%, is statistically significant at 0.01 level.

Degradation of speech recognition performance over lossy data networks

In this work we investigate the effects of lossy data networks on the speech recognition performance, utilizing a stock information corpus.

Handling Ellipsis in a Spoken Medical Phraselator

We consider methods for handling incomplete (elliptical) utterances in spoken phraselators, and describe how they have been implemented inside BabelDr, a substantial spoken medical phraselator. The challenge is to extend the phrase matching process so that it is sensitive to preceding dialogue context. We contrast two methods, one using limited-vocabulary strict grammar-based speech and language processing and one using large-vocabulary speech recognition with fuzzy grammar-based processing, and present an initial evaluation on a spoken corpus of 821 context-sentence/elliptical-phrase pairs. The large-vocabulary/fuzzy method strongly outperforms the limited-vocabulary/strict method over the whole corpus, though it is slightly inferior for the subset that is within grammar coverage. We investigate possibilities for combining the two processing paths, using several machine learning frameworks, and demonstrate that hybrid methods strongly outperform the large-vocabulary/fuzzy method.

Constructing Knowledge-Based Feedback in the Context of an Interactive Spoken CALL Application

Modern Computer Assisted Language Learning (CALL) systems use speech recognition to give students the opportunity to build up their spoken language skills through interactive practice with a mechanical partner. Besides the obvious benefits that these systems can offer, e.g. flexible and inexpensive learning, user interaction in this context can often be problematic. In this article, the authors introduce a parallel layer of feedback in a CALL application, which can monitor interaction, report errors and provide advice and suggestions to students. This mechanism combines knowledge accumulated from four different inputs in order to decide on appropriate feedback, which can be customized and adapted in terms of phrasing, style and language. The authors report the results from experiments conducted at six lower secondary classrooms in German-speaking Switzerland with and without this mechanism. After analyzing approximately 13,000 spoken interactions it can be reasonably argued that their parallel feedback mechanism in L2 actually does help students during interaction and contributes as a motivation factor.

Rapid Construction of a Web-Enabled Medical Speech to Sign Language Translator Using Recorded Video

We describe an experiment in which sign-language output in Swiss French Sign Language (LSF-CH) and Australian Sign Language (Auslan) was added to a limited-domain medical speech translation system using a recorded video method. By constructing a suitable web tool to manage the recording procedure, the overhead involved in creating and manipulating the large set of files involved could be made easily manageable, allowing us to focus on the interesting and non-trivial problems which arise at the translation level. Initial experiences with the system suggest that the recorded videos, despite their unprofessional appearance, are readily comprehensible to Deaf informants, and that the method is promising as a simple short-term solution for this type of application.