Nobuhiro Furuyama

Prolegomena of a theory of between-person coordination of speech and gesture

This paper describes some examples of what is here called between-person coordination of speech and gesture. It attempts to explain the phenomenon in terms of the principles set forth in Growth Point theory (GP theory). McNeill has proposed this to explain what is here contrastively called within-person coordination of speech and gesture. In due course, the advantages of GP theory over the other existing theories of speechgesture coordination will be discussed, and some implications for interactional graphical communication will be noted.

Utility of Gestural Cues in Indexing Semantic Miscommunication

In multimedia data analysis, automated indexing of conversational video is an emerging topic. One challenging problem in this topic is the recognition of higher-level concepts, such as \emph{miscommunications} in conversations. While detecting these miscommunications is generally easy for the speakers as well as for observers, it is not currently understood which cues contribute to their detection and to what extent. We investigate the possibility of indexing the occurrence of miscommunications in psychotherapeutic face-to-face conversations from gestural patterns. The applicability of machine learning is investigated as a means of detecting miscommunication from gestural patterns observed in the conversations. Both simple and complex classifiers are constructed using different features taken from the gesture data. Both short-term and long-term effects are tested using different time window sizes. Also, two types of gestures, communicative and non-communicative, are considered. The experimental results suggest that there \emph{does not exist a single gestural feature} that can explain the occurrence of semantic miscommunication. Another interesting finding is that gestural cues correlate more with long-term gestural patterns than short-term ones.

Gestural cue analysis in automated semantic miscommunication annotation

The automated annotation of conversational video by semantic miscommunication labels is a challenging topic. Although miscommunications are often obvious to the speakers as well as the observers, it is difficult for machines to detect them from the low-level features. We investigate the utility of gestural cues in this paper among various non-verbal features. Compared with gesture recognition tasks in human-computer interaction, this process is difficult due to the lack of understanding on which cues contribute to miscommunications and the implicitness of gestures. Nine simple gestural features are taken from gesture data, and both simple and complex classifiers are constructed using machine learning. The experimental results suggest that there is no single gestural feature that can predict or explain the occurrence of semantic miscommunication in our setting.

Comparing intra- and inter-personal coordination systems: Perceptual effect on stability of finger tapping movement

An experiment was conducted to reveal the differences between intra-personal and inter-personal systems in terms of the perceptual effect on the stability of the finger-tapping movement in the in-phase vs. anti-phase modes. We investigated a between-subjects factor, i.e., the intra-/inter-personal condition, and a within-subject factor, i.e., the phase mode. In the intra-personal condition, ten participants each bimanually tapped their index fingers on a table, in time with a metronome, at a frequency gradually increasing from 1 to 3 Hz over a 30-s trial. In the inter-personal condition, ten pairs of participants were asked to perform the same task as in the intra-personal condition, but to use their right or left index finger, while sitting next to each other and looking at each others fingers moving. The analysis showed that the average number of phase transitions and average time-to-transition differed between the intra- and inter-personal systems. Some of the results do not agree with the predictions made using the theoretical model proposed in previous studies on inter-limb coordination.

Can Hermit Crabs Perceive Affordance for Aperture Crossing

An animal’s perception of its body size is modified when it adapts its body to changes in a complex environment. This ability is essential for animals that use tools or cross apertures. Here, we show that terrestrial hermit crabs, Coenobita rugosus, which frequently change shells, can perceive the width of the aperture to be crossed, dependent on the shape of their shells. Hermit crabs walked in a corridor that had two different size apertures; most of the crabs with a large shell did not cross the narrow aperture, indicating an awareness of aperture width. Moreover, most of the crabs with a small shell with an attachment did not select the narrow aperture, either. These results are the first demonstration of animals perceiving affordance while carrying objects.

Multimodality in Multispace Interaction (MiMI)

We held International Workshop on Multimodality in Multispace Interaction (MiMI), at Sunport Hall Takamatsu, Takamatsu City, Kagawa prefecture in Japan on December 1-2, 2011. The workshop was part of JSAI International Symposia on Artificial Intelligence (JSAI-isAI 2011) sponsored by the Japanese Society for Artificial Intelligence. All he papers collected here were presented at the workshop either as invited talks or as accepted papers . Incorporating discussions, comments, and questions, workshop presenters revised their papers and submit them to this proceedings . The submitted papers were peer-reviewed once again and three out of the eight papers were accepted in the end. Our special gratitude goes to the anonymous reviewers of the papers for their dedicated efforts to make very constructive and useful comments for the authors to make their papers more convincing and intriguing. Before we proceed to the papers themselves, we would like to introduce the readers to the aims and scope of MiMI 2011, by showing a piece of memo that we made in preparation of a proposal of the workshop to JSAI.

Temporal development of pragmatics and dynamics in conversation for building consensus

We studied the process of information integration through observing pragmatics and dynamics of conversations toward a consensus between two persons. From the pragmatic perspective, the temporal development of pragmatics regarding their consensus was analyzed with conversational analysis and subjective evaluation. From the dynamical perspective, the temporal development of response time of utterance was analyzed with correlation analysis. As results, their typical developments were clarified. Pragmatic stages regarding consensus were taking turns alternately toward a consensus. The subjective evaluated values on their consensus increased toward the end of the conversation. The temporal variation of response times were synchronized interpersonally toward the end of the conversation. These results show that temporal development of pragmatics and dynamics are mutually related. Furthermore, it is suggested that synchronization dynamics of utterance plays an important role for information integration of consensus building.