Mika Enomoto

When should animated agents give additional instructions to users? - Monitoring user’s understanding in multimodal dialogues -

In multimodal communication, verbal and nonverbal behaviors such as gestures and manipulating objects in a workspace occur in parallel, and are coordinated in proper timing to each other. This paper focuses on the interaction between a beginner user using a video recorder application on PC and a multimodal animated help agent, and presents a probabilistic model of fine-grained timing dependencies among different behaviors of different modalities. First, we collect user-agent dialogues using a Wizard-of-Oz experimental setting, and then the collected verbal and nonverbal behavior data will be used to build a Bayesian network model, which can predict the likelihood of successful mouse clicks in near future, given evidence associated with the status of speech, agents gestures and users mouse actions. Finally, we attempt to determine proper timing when the agent should give additional instructions by estimating the likelihood of a mouse click occurrence.

Simultaneous prediction of dialog acts and address types in three-party conversations

This paper reports on automatic prediction of dialog acts and address types in three-party conversations. In multi-party interaction, dialog structure becomes more complex compared to one-to-one case, because there is more than one hearer for an utterance. To cope with this problem, we predict dialog acts and address types simultaneously on our framework. Prediction of dialog act labels has gained to 68.5% by considering both context and address types. CART decision tree analysis has also been applied to examine useful features to predict those labels.

Experimental investigation of end-of-utterance perception by final lowering in spontaneous Japanese

It is considered that final lowering, which is a phenomenon that the fundamental frequencies (F0s) fall significantly in sentence-final position, marks the end of an utterance in Japanese. However, it is not clear whether the hearer utilizes final lowering to recognize the end of the utterance. In this paper we investigated effect of final lowering on the perception of an end-of-utterance in spontaneous Japanese speech. We adopt a Bayesian hierarchical model to estimate final accentual phrases or utterance-final elements related to the end of the utterance from prosodic features. Results suggested that final lowering was associated with the syntactic elements mostly placed at the end of a Japanese utterance, but not with the end-of-utterance itself. We, then, carried out perceptual experiments using Japanese utterances with modified F0s either in the middle or at the end. The results showed that the subjects detected the end of the utterance even in the absence of final lowering and were not affected by the presence of an F0 downstep simulating the final lowering in the middle of the utterance. That suggests that Japanese hearers do not use final lowering to perceive the end of an utterance.

Implicit Proposal Filtering in Multi-Party Consensus-Building Conversations

An attempt was made to statistically estimate proposals which survived the discussion to be incorporated in the final agreement in an instance of a Japanese design conversation. Low level speech and vision features of hearer behaviors corresponding to aiduti, noddings and gaze were found to be a positive predictor of survival. The result suggests that non-linguistic hearer responses work as implicit proposal filters in consensus building, and could provide promising candidate features for the purpose of recognition and summarization of meeting events.

A study on prediction of end-of-utterance by prosodic features and phrase-dependency structure in spontaneous speech

This study is aimed at predicting the end of utterance by prosodic features and syntactic structure for spontaneous speech. In spontaneous everyday conversation, participants must predict the ends of utterances of a speaker to perform smooth turn-taking. We consider that they utilize not only syntactic factors but also prosodic factors for the end-of-utterance prediction because of the difficulty of prediction of a syntactic completion point in spontaneous Japanese speech. In previous studies, it was observed that prosodic factors changed such that the general fundamental frequency of utterance declined gradually toward the end of an utterance, and the intensity decreased significantly in the final accentual phrase. However, it is not clear what prosodic features support the prediction. We focused on dependency structure among bunsetsu-phrases as the syntactic factor and investigated the relation between the phrase-dependency and prosodic features based on a spontaneous Japanese conversation corpus. The r...

Laughter around the End of Storytelling in Multi-party Interaction

This research argues that the sequence organization around the end of a storytelling is affected by which of the hearers inserts a laughter to which position in the sequence. Laughters before the beginning point of possible completion of storytelling never affect on the sequence organization, while those which occur within a transition space change who is to speak first after the storytelling. Though an oriented recipient (OR) is a default candidate who can speak first after the storytelling, laughs within a transition space let the OR pass his/her own occasion for speech to another participant.

Unsupervised clustering in multimodal multiparty meeting analysis

Nonverbal signals such as gazes, head nods, facial expressions, and bodily gestures play significant roles in organizing human interactions. Their significance is even more emphasized in multiparty settings, since many interaction organization behaviors, for example, turn-taking and participation role assignment, are realized nonverbally. Several projects have been involved in collecting multimodal corpora [3,4] for multiparty dialogues, to develop techniques for meeting event recognitions from nonverbal as well as verbal signals (e.g., [11,2]).

人-人,人-ヒューマンエージェントの社会的インタラクションにおける言語・非言語行為の配置規則( ソーシャルインテリジェンス)

本研究では，エージェントを相手にした時のインタラクション方略に人相手のときとの異同があることを示し，人間の行動モデルをベースにヒューマンエージェントを実装するとき，考慮すべき人間のインタラクション方略のあることを提案する．ここでは，パソコン操作課題における人と人，人とエージェントの対話を素材として，どのように言語・非言語行為がインタラクションの中で配置されるのかを分析することで，エージェントに対したときに選択される方略を明らかにする．まず，人対人と人対エージェント対話の基礎的特徴を観察し，人対エージェントの対話では人の発話量が少なく，相づちや応答が稀にしか差し挟まれないことを示す．次に，非言語行為を含めた人対人の行為の配置規則を定式化し，人対エージェントのインタラクションにおいてこの規則がどのように破られるかを示す．そして，この違反が，相づちや応答の変わりに，相手発話への理解を示すためになされた補償的行為であることを明らかにする．本研究では，エージェントを相手にした時のインタラクション方略に人相手のときとの異同があることを示し，人間の行動モデルをベースにヒューマンエージェントを実装するとき，考慮すべき人間のインタラクション方略のあることを提案する．ここでは，パソコン操作課題における人と人，人とエージェントの対話を素材として，どのように言語・非言語行為がインタラクションの中で配置されるのかを分析することで，エージェントに対したときに選択される方略を明らかにする．まず，人対人と人対エージェント対話の基礎的特徴を観察し，人対エージェントの対話では人の発話量が少なく，相づちや応答が稀にしか差し挟まれないことを示す．次に，非言語行為を含めた人対人の行為の配置規則を定式化し，人対エージェントのインタラクションにおいてこの規則がどのように破られるかを示す．そして，この違反が，相づちや応答の変わりに，相手発話への理解を示すためになされた補償的行為であることを明らかにする．