Friedrich Neubarth

Modeling and interpolation of Austrian German and Viennese dialect in HMM-based speech synthesis

An HMM-based speech synthesis framework is applied to both standard Austrian German and a Viennese dialectal variety and several training strategies for multi-dialect modeling such as dialect clustering and dialect-adaptive training are investigated. For bridging the gap between processing on the level of HMMs and on the linguistic level, we add phonological transformations to the HMM interpolation and apply them to dialect interpolation. The crucial steps are to employ several formalized phonological rules between Austrian German and Viennese dialect as constraints for the HMM interpolation. We verify the effectiveness of this strategy in a number of perceptual evaluations. Since the HMM space used is not articulatory but acoustic space, there are some variations in evaluation results between the phonological rules. However, in general we obtained good evaluation results which show that listeners can perceive both continuous and categorical changes of dialect varieties by using phonological transformations employed as switching rules in the HMM interpolation.

Towards context-aware personalization and a broad perspective on the semantics of news articles

We analyze preferences and the reading flow of users of a popular Austrian online newspaper. Unlike traditional news filtering approaches, we postulate that a users preference for particular articles depends not only on the topic and on propositional contents, but also on the users current context and on more subtle attributes. Our assumption is motivated by the observation that many people read newspapers because they actually enjoy the process. Such sentiments depend on a complex variety of factors. The present study is part of an ongoing effort to bring more advanced personalization to online media. Towards this end, we present a systematic evaluation of the merit of contextual and non-propositional features based on real-life clickstream and postings data. Furthermore, we assess the impact of different recommendation strategies on the learning performance of our system.

Social evaluation of artificial agents by language varieties

In Sociolinguistics, language attitude studies based on natural voices have provided evidence that human listeners socially assess and evaluate their communication partners according to the language variety they use. Similarly, research on intelligent agents has demonstrated that the degree an artificial entity resembles a human correlates with the likelihood that the entity will evoke social and psychological processes in humans. Taking the two findings together, we hypothesize that synthetically generated language varieties have social effects similar to those reported from language attitude studies on natural speech. We present results from a language-attitude study based on three synthetic varieties of Austrian German. Our results on synthetic speech are in accordance with previous findings from natural speech. In addition, we show that language variety together with voice quality of the synthesized speech bring about attributions of different social aspects and stereotypes and influence the attitudes of the listeners toward the artificial speakers.

Towards a context-sensitive online newspaper

We give a detailed account of our experiences in implementing a personalized online newspaper that draws---among other hints---on the context of the user. At the algorithmic core of our framework lies a machine learning model that incorporates numerous features of the eligible articles and the users current situation. Some of the most important design decisions, however, concern the presentation of suggestions, the collection of explicit and implicit feedback, as well as diversity of the recommendations. We present numerical results obtained during the pilot phase of the project that address a number of these concerns and end with a discussion of open questions and future directions.

A Hybrid Approach to Statistical Machine Translation Between Standard and Dialectal Varieties

Using statistical machine translation (SMT) for dialectal varieties usually suffers from data sparsity, but combining word-level and character-level models can yield good results even with small training data by exploiting the relative proximity between the two varieties. In this paper, we describe a specific problem and its solution, arising with the translation between standard Austrian German and Viennese dialect. In general, for a phrase-based approach to SMT, complex lexical transformations and syntactic reordering cannot be dealt with satisfyingly. In a situation with sparse resources it becomes merely impossible. These are typical cases where rule-based preprocessing of the source data is the preferable option, hence the hybrid character of the resulting system. One such case is the transformation between synthetic imperfect verb forms to perfect tense with finite auxiliary and past participle, which involves detection of clause boundaries and identification of clause type. We present an approach that utilizes a full parse of the source sentences and discuss the problems that arise using such an approach. Within the developed SMT system, the models trained on preprocessed data unsurprisingly fare better than those trained on the original data, but also unchanged sentences gain slightly better scores. This shows that introducing a rule-based layer dealing with systematic non-local transformations increases the overall performance of the system, most probably due to a higher accuracy in the alignment.

Design and development of spoken dialog systems incorporating speech synthesis of Viennese varieties

This paper describes our work on the design and development of a spoken dialog system, which uses synthesized speech of various different Viennese varieties. In a previous study we investigated the usefulness of synthesis of varieties. The developed spoken dialog system was especially designed for the different personas that can be realized with multiple varieties. This brings more realistic and fun-to-use spoken dialog systems to the end user and can serve as speech-based user interface for blind users and users with visual impairment. The benefits for this group of users are the increased acceptability and also comprehensibility that comes about when the synthesized speech reflects the users linguistic and/or social identity.

Optimizing phonetic encoding for viennese unit selection speech synthesis

While developing lexical resources for a particular language variety (Viennese), we experimented with a set of 5 different phonetic encodings, termed phone sets, used for unit selection speech synthesis. We started with a very rich phone set based on phonological considerations and covering as much phonetic variability as possible, which was then reduced to smaller sets by applying transformation rules that map or merge phone symbols. The optimal trade-off was found measuring the phone error rates of automatically learnt grapheme-to-phone rules and by a perceptual evaluation of 27 representative synthesized sentences. Further, we describe a method to semi-automatically enlarge the lexical resources for the target language variety using a lexicon base for Standard Austrian German.

Extension of the Action Verb Corpus for Supervised Learning

The Action Verb Corpus (AVC) is a multimodal dataset of simple actions for robot learning. The extension introduced here is especially geared to supervised learning of actions from human motion data. Recorded are RGB-D videos of the test scene, grayscale videos from the user’s perspective, human hand trajectories, object poses and speech utterances. The three actions TAKE, PUT and PUSH are annotated with labels for the actions in different granularity.

Speak to me and I tell you who you are! A language-attitude study in a cultural-heritage application

Research on intelligent agents has demonstrated that the degree an artificial entity resembles a human correlates with the likelihood that the entity will evoke social and psychological processes in humans. Language-attitude studies based on natural voices have provided evidence that human listeners socially assess and evaluate their communication partners according to the language variety they use. Taking the two findings together, we hypothesize that synthetically generated language varieties have social effects similar to those reported from language-attitude studies on natural speech. We present the design of a set of synthetic voices representing standard and dialectal varieties of Austrian German which were built into an existing cultural-heritage application letting virtual tourist guides speak in different varieties. With this setup, we performed a language-attitude study assessing the social evaluation of the characters represented by the synthetic voices. Our results are in accordance with previous findings from natural speech, but it also turned out that the specific context constitutes a major criterion for the preference or rejection of certain language varieties. In addition, we show that not only the particular variety, but also features relating to the voice quality of the synthesized speech bring about attributions of different social aspects and stereotypes. Together they strongly influence the attitudes of the listeners towards the artificial speakers showing the importance of an accurate voice design—including features related to particular language varieties—for the development of artificial agents.

A Distributional Concept for Modeling Dialectal Variation in TTS

Current TTS systems usually represent a certain standard of a given language, regional or social variation is barely reflected. In this paper, we describe certain strategies for modeling language varieties on the basis of a common language resource, in particular Austrian varieties from German sources. The goal is to find optimal procedures in order to represent these differences with minimal efforts in annotation and processing. We delimit the discussion to the lower levels of the transformation of linguistic information --- phonetic encoding. One question is if it is necessary or desirable to aim at maximal accurateness of the phonetic transcriptions. We will show that while certain differences could in principle be captured by the context within the speech data, other differences definitely have to be re-modelled, since they either involve ambiguous correspondences, or the string of phones is different in such a way that automatic procedures such as alignment or unit selection would be negatively affected, hence degrade the overall quality of the synthesized speech.