Yoko Greenberg

Communicative prosody generation using language common features provided by input lexicons

We already examined language independent control characteristics of the communicative prosody generation using multi-dimensional impressions of input lexicons. In this paper, we synthesized English single phrase utterances using prosodic characteristics of Japanese speech aiming at language independent applications. The reading-style speech prosodies of English phrases were modified by prosodic characteristics derived from one-word utterance of Japanese speech “n”. Modifications were carried out based on lexical impressions corresponding to six impressions consisting of confident, doubtful, allowable, unacceptable, positive and negative. The perceptual evaluation experiment showed the naturalness of speech with communicative prosody modified by the impression of input lexicons. These experimental results support the usefulness of the communicative prosody control based on the impression of input lexicons and suggest possibilities of language independent applications.

Global F0 control parameter prediction based on impressions for communicative prosody generation

Aiming at communicative speech synthesis, prosody control using impressions has been proposed by applying the correlation between impressions of input lexicons and prosody. In this paper, as the first step to compute communicative prosody, we attempt to predict the F0 generation model parameters by estimating the impressions of input sentence from its constituent lexicons. To obtain an impression vector consisting of three dimensional factors (positive-negative, confident-doubtful and allowable-unacceptable) for a given input utterance, we proposed a computational scheme to calculate impression vectors using impression scores of constituent words. Using obtained sentence impression vectors, F0 control parameters are predicted by applying three-layered feed-forward neural networks. To evaluate the effectiveness of the proposed computational framework, we experimentally confirmed that F0 parameters of communicative speech could be generated from the impressions of input lexicons.

Sentiment analysis of color attributes derived from vowel sound impression for multimodal expression

In this paper, the sentiment correlation between color and speech has been investigated aiming at multimodal expression of information embedded in speech. To cope with the necessity of information specification for further analyses and control of speech so called para-linguistic information, we carried out an sentiment experiment on the correlations between speech characteristics and color attributes associated by perceptual impressions. Using five vowel categories with twelve different prosodie patterns, we found high correlations between speech features (vowel categories and FO) and three attributes of color (hue, saturation and value). These correlations open up wide possibilities to describe and quantitatively analyze speech information that has not yet been systematically studied but partially treated as emotional variants or para-linguistic information.

Communicative Speech Synthesis as Pan-Linguistic Prosody Control

A series of prosody analyses have been carried out to find the correlations between lexical attributes and communicative prosody variations. By employing lexical attributes showing their impressions, Multi-Dimensional Scaling (MDS) has revealed that three-dimensional perceptual impression space (positive–negative, confident–doubtful, allowable–unacceptable) nicely correlates to F0 heights and their dynamics. Based on these correlations, a communicative prosody generation scheme is newly proposed for input text based on impression attributes of input lexicons using the command–response model in a pan–linguistic language common framework. In this scheme, the communicative component derived by constituent lexicons is added to conventional reading style prosody. This communicative prosody generation formalism not only provides the first methodology for communicative prosody generation, but also creates a new linguistic research paradigm where traditionally abandoned notions like paralinguistics will become linguistically essential core items for communicative spoken language science.

On the prosody control characteristics of nonverbal utterances and its application to communicative prosody generation

In this paper, prosodic characteristics of nonverbal utterances were analyzed using an F0 generation model proposed by Fujisaki aiming at communicative speech generation. From the analysis, the different distributions of F0 generation parameters have been observed for prototypical four dynamic patterns (rise, gradual fall, fall, and rise&down). Since former works have shown that these differences can correspond to their impressions (such as confident‐doubtful, allowable‐unacceptable, and positive‐negative) expressed by multi‐dimensional vectors, we tried to make a computational model from impression vector to F0 generation parameters. By employing a statistical optimization technique, we have achieved the mapping from impression vectors to prosody generation parameters. Perceptual evaluation tests using neutral words have confirmed the effectiveness of the mapping to provide communicative speech prosody. [Work supported in part by Waseda Univ. RISE research project of ‘‘Analysis and modeling of human mech...