Koki Ijuin

Multimodal corpus of multiparty conversations in L1 and L2 languages and findings obtained from it

To investigate the differences in communicative activities by the same interlocutors in Japanese (their L1) and in English (their L2), an 8-h multimodal corpus of multiparty conversations was collected. Three subjects participated in each conversational group, and they had conversations on free-flowing and goal-oriented topics in Japanese and in English. Their utterances, eye gazes, and gestures were recorded with microphones, eye trackers, and video cameras. The utterances and eye gazes were manually annotated. Their utterances were transcribed, and the transcriptions of each participant were aligned with those of the others along the time axis. Quantitative analyses were made to compare the communicative activities caused by the differences in conversational languages, the conversation types, and the levels of language expertise in L2. The results reveal different utterance characteristics and gaze patterns that reflect the differences in difficulty felt by the participants in each conversational condition. Both total and average durations of utterances were shorter in their L2 than in their L1 conversations. Differences in eye gazes were mainly found in those toward the information senders: Speakers were gazed at more in their second-language than in their native-language conversations. Our findings on the characteristics of conversations in the second language suggest possible directions for future research in psychology, cognitive science, and human–computer interaction technologies.

Difference in Eye Gaze for Floor Apportionment in Native- and Second-Language Conversations

In face-to-face communication, eye gaze is known to play various roles such as managing the attention of interlocutors, expressing intimacy, exercising social control, highlighting particular speech content, and coordinating floor apportionment. For second language (L2) communication, one’s perception of eye gaze is expected to have more importance than for native language (L1) because eye gaze is assumed to partially compensate for the deficiencies of verbal expressions. This paper examines and clarifies the efficiency of the function of eye gaze in the coordination of floor apportionment through quantitative analyses of eye gaze during three-party conversations in L1 and L2. Specifically, the authors conducted ANOVA tests on the eye-gaze statistics of a speaker and two listeners during utterances while focusing on whether floor-switch occurs subsequent to the utterance. The analysis results show that the listener who is gazed at more by the speaker is more likely to be the next speaker with a higher probability in L2 than in L1 conversations. Meanwhile, the listeners gaze more at the speaker in L2 than in L1 conversation for both the utterances just before a floor switch and cases with no floor switch. These results support the observation that the eye gaze of the speaker is efficient for floor apportionment in L2 conversations and suggest that longer listeners’ eye gazes in L2 conversations also function efficiently in smooth floor apportionment.

Eye Gaze Analyses in L1 and L2 Conversations: From the Perspective of Listeners' Eye Gaze Activity

The importance of conversation in a second language (L2) during international collaboration continues to increase, but the features of non-verbal communications such as eye gaze and gestures in L2 conversation have not been clarified to the extent they have in the native language (L1). This study provides quantitative analyses of eye gaze activities to examine their differences in conversations between L1 and L2. These analyses clarify that the frequency of utterances during which listeners gaze at speakers throughout the utterance is greater in L2 than in L1 conversations when they join small-party conversations.

Floor Apportionment and Mutual Gazes in Native and Second-Language Conversation

Quantitative analysis of gazes between a speaker and listeners was conducted from the viewpoint of mutual activities in floor apportionment, with the assumption that mutual gaze plays an important role in coordinating speech interaction. We conducted correlation analyses of the speakers and listeners gazes in a three-party conversation, comparing native language (L1) and second language (L2) interaction in two types (free-flowing and goal-orient- ed). The analyses showed significant correlations between gazes from the current to the next speaker and those from the next to the current speaker during utterances preceding a speaker change in L1 conversation, suggesting that the participants were coordinating their speech turns with mutual gazes. In L2 conversation, however, such a correlation was found only in the goal-oriented type, suggesting that linguistic proficiency may affect the floor-apportionment function of mutual gazes, possibly because of the cognitive load of understanding/producing utterances.

Comparison of Effect of Speaker's Eye Gaze on Selection of Next Speaker between Native- and Second-Language Conversations

In face-to-face communication, eye gaze is known to play various roles such as managing the attention of interlocutors, expressing intimacy, exercising social control, highlighting particular speech content and coordinating floor apportionment. In second language (L2) communication, one’s perception of eye gaze is expected to have more importance than in native language (L1) because eye gaze can be used to partially compensate for the deficiencies of verbal expressions. This paper examines the efficiency of eye gaze for floor apportionment through quantitative analyses of eye gaze during three-party conversations in L1 and L2. The authors analyze the average ratios at which the participant to whom the speaker gazes takes the floor according to the duration of pauses between two consecutive utterances. The analysis results show that this ratio decreases as the duration of a pause becomes longer in L1 conversations, whereas the gazed-at participant often takes the floor even after a longer duration of pause in L2 conversations. This suggests that the effect of the speaker’s eye gaze decreased when the duration of pause was prolonged in L1 conversations, whereas this effect was maintained in L2 conversations.

Multimodal Japanese corpus of multi-party conversation on two different topic types

We collected a multimodal corpus of three-party conversation in Japanese (mother tongue) and in English (second language) to compare the basic characteristics of conversation, such as utterance duration and silence, for two topic types: free-flowing and task-oriented. The corpus contains video and audio data of eighty conversations (four hundred eighty minutes in total) with speech duration, upper body motion, and eye gaze activity labels. This article reports on the corpus collecting process and the results of preliminary analyses.