Takaaki Shochi

Multimodal Indices to Japanese and French Prosodically Expressed Social Affects

Whereas several studies have explored the expression of emotions, little is known on how the visual and audio channels are combined during production of what we call the more controlled social affects, for example, “attitudinal” expressions. This article presents a perception study of the audovisual expression of 12 Japanese and 6 French attitudes in order to understand the contribution of audio and visual modalities for affective communication. The relative importance of each modality in the perceptual decoding of the expressions of four speakers is analyzed as a first step towards a deeper comprehension of their influence on the expression of social affects. Then, the audovisual productions of two speakers (one for each language) are acoustically (F0, duration and intensity) and visually (in terms of Action Units) analyzed, in order to match the relation between objective parameters and listeners perception of these social affects. The most pertinent objective features, either acoustic or visual, are then discussed, in a bilingual perspective: for example, the relative influence of fundamental frequency for attitudinal expression in both languages is discussed, and the importance of a certain aspect of the voice quality dimension in Japanese is underlined.

Comparison between Japanese children and adults perception of prosodic politeness expressions

Previous work examined the contribution of audio and visual modalities for perception of Japanese social affects by adults. The results showed audio and visual information contribute to the perception of culturally encoded expressions, and show an important synergy when presented together. Multimodal presentation allows foreign adult listeners to recognize culturally encoded expressions of Japanese politeness which they cannot recognize with an audio‐only stimuli. This current work analyzes the recognition performance of politeness expressions by Japanese children 13 to 14 years old. Stimuli, based on one sentence with an affectively neutral meaning, are performed with five different expressions of politeness. Subjects listen three times to each stimulus and judge the intended message of the speaker. The stimuli are presented as audio‐only, visual‐only, audio‐visual. Listeners rate the social status of the hearer and the degree of politeness on a nine‐point scale ranging from polite to impolite. The resul...

Because attitudes are social affects, they can be false friends...

The attitudes of the speaker during a verbal interaction are affects built by language and culture. Since they are a sophisticated material for expressing complex affects, using a channel of control that is surely not confused with emotions, they are the larger part of the affects expressed during an interaction, as it could be shown on large databases by Campbell [3]. strong Twelve representative attitudes of Japanese are given to be listened both by Japanese native speaker and French native speaker naive in Japanese. They include “doubt-incredulity”, “evidence”, “exclamation of surprise”, “authority”, “irritation”, “admiration”, “declaration”, “interrogation”, and four socially referenced degrees of politeness: “simple politeness”, “sincerity-politeness”, “kyoshuku” and “arrogance-impoliteness” (Sadanobu [11]). Two perception experiments using a closed forced choice were carried out, each attitude being introduced by a definition and some examples of real situations. The 15 native Japanese subjects discriminate all the attitudes over chance, with some little confusion inside the politeness class. French subjects do not process the concept of degree of politeness: they do not identify the typical Japanese politeness degrees. The prosody of “kyoshuku”, highest degree of politeness in Japanese, is misunderstood by French on contrary meaning as “impoliteness”, “authority” and “irritation”.

Formant lowering in spontaneous crying speech.

Acoustic and articulatory recordings were made at the EMA facilities of NTT Research Laboratories, Atsugi, Japan, for an American English speaker producing (a) spontaneous crying speech and (b) imitation of phrasing of the original crying speech, as control data. Articulatory analysis indicates differences in jaw, lip, and tongue positions for crying speech versus control speech. Acoustic analysis also shows that for crying speech compared with control speech, not only F0 increases but also higher formants tend to be lowered. Results of perception tests using the copy‐synthesis program STRAIGHT (Kawahara) to morph a continuum of stimuli, keeping F0, duration, and intensity constant, suggest listeners to use cues of lowered formants to perceive emotional intensity of an utterance. Recent biophysiological modeling studies suggest that lowered formants may be due to a lowered larynx along with an expanded hypopharyngeal region [e.g., D. Honda, J. Acoust. Soc. Am. (1966); Kitamura et al., Acoust. Sci. Tech. (...