Ichiro Umata

Graphical representation in graphical dialogue

Abstract This paper explores the influence of communicative interaction on the form of graphical representations. A referential communication task is described which involves exclusively graphical dialogue. In this task subjects communicate about pieces of music by drawing. The drawings produced fall into two basic types: Abstract and Figurative. Three hypotheses are developed about the factors influencing the use of these drawing types: efficiency of production, suitability for the task and level of communicative interaction. Experimental evidence is presented which indicates that the drawing types do not differ in the amount of effort required to produce them. The results indicate that (1) Abstract drawings are more effective than Figurative drawings for comparative tasks and (2) a key constraint on their use is level of direct communicative interaction. It is argued that these observations result from differences in the underlying semantic models of music associated with the drawing types and the consequences these differences have for communicative coordination.

Difference in Eye Gaze for Floor Apportionment in Native- and Second-Language Conversations

In face-to-face communication, eye gaze is known to play various roles such as managing the attention of interlocutors, expressing intimacy, exercising social control, highlighting particular speech content, and coordinating floor apportionment. For second language (L2) communication, one’s perception of eye gaze is expected to have more importance than for native language (L1) because eye gaze is assumed to partially compensate for the deficiencies of verbal expressions. This paper examines and clarifies the efficiency of the function of eye gaze in the coordination of floor apportionment through quantitative analyses of eye gaze during three-party conversations in L1 and L2. Specifically, the authors conducted ANOVA tests on the eye-gaze statistics of a speaker and two listeners during utterances while focusing on whether floor-switch occurs subsequent to the utterance. The analysis results show that the listener who is gazed at more by the speaker is more likely to be the next speaker with a higher probability in L2 than in L1 conversations. Meanwhile, the listeners gaze more at the speaker in L2 than in L1 conversation for both the utterances just before a floor switch and cases with no floor switch. These results support the observation that the eye gaze of the speaker is efficient for floor apportionment in L2 conversations and suggest that longer listeners’ eye gazes in L2 conversations also function efficiently in smooth floor apportionment.

Using Graphics to Communicate Across Cultures

This paper explores the potential of graphical communication for cross-linguistic/cultural interaction. Results demonstrate that interactive graphical communication provides a useful cross-cultural communication tool. However, communicative success is limited by information type, such that concepts that do not share a comparable visual form across cultures are less successfully communicated.

Speech and Graphical Interaction in Multimodal Communication

Graphical communications, such as dialogues using maps, drawings, or pictures, provide people with two independent modalities through which they can interact with each other. In such conversation, graphical interaction can be both sequential and parallel, affected by the activity-dependent constraints imposed by the task performed in the interaction (Umata, Shimojima, Katagiri, and Swoboda (2003)). In this paper, we compare the patterns of speech and graphical interaction in collaborative problem-solving tasks. The amount of speech overlap did not show a significant difference among four task conditions, although graphical overlaps did. This shows that both resource-dependent and activity-dependent constraints play a significant role in determining the interaction organization.

Floor Apportionment and Mutual Gazes in Native and Second-Language Conversation

Quantitative analysis of gazes between a speaker and listeners was conducted from the viewpoint of mutual activities in floor apportionment, with the assumption that mutual gaze plays an important role in coordinating speech interaction. We conducted correlation analyses of the speakers and listeners gazes in a three-party conversation, comparing native language (L1) and second language (L2) interaction in two types (free-flowing and goal-orient- ed). The analyses showed significant correlations between gazes from the current to the next speaker and those from the next to the current speaker during utterances preceding a speaker change in L1 conversation, suggesting that the participants were coordinating their speech turns with mutual gazes. In L2 conversation, however, such a correlation was found only in the goal-oriented type, suggesting that linguistic proficiency may affect the floor-apportionment function of mutual gazes, possibly because of the cognitive load of understanding/producing utterances.

Effects of the mode of tempo change on perception of tempo change

Tempo is one of the basic factors in musical expression. Although there are studies on perception of tempo change, little is known about how the mode of tempo change affects sensitivity to the change. In this paper, we analyze the effects of modes of tempo change on perception of tempo change. Our analysis focuses on sensitivity to tempo change. Forty-six subject participants were divided into three groups according to their musical experience and the type of playing they are used to. ((A) 15 inexperienced, (B) 21 pianists mostly playing solo, (C) 10 players of musical instruments other than piano mostly playing in groups). We used synthetic piano single tone sequences that change tempo gradually from the common initial value to various target values as stimuli. We also manipulated the mode of tempo change: linear, exponential, and their average. Subject participants were asked to indicate the time point of perception by pressing a key as soon as they perceived the tempo change. Contrary to our presumptio...

Multimodal Japanese corpus of multi-party conversation on two different topic types

We collected a multimodal corpus of three-party conversation in Japanese (mother tongue) and in English (second language) to compare the basic characteristics of conversation, such as utterance duration and silence, for two topic types: free-flowing and task-oriented. The corpus contains video and audio data of eighty conversations (four hundred eighty minutes in total) with speech duration, upper body motion, and eye gaze activity labels. This article reports on the corpus collecting process and the results of preliminary analyses.