Patrick Jeuniaux

Behavior matching in multimodal communication is synchronized

A variety of theoretical frameworks predict the resemblance of behaviors between two people engaged in communication, in the form of coordination, mimicry, or alignment. However, little is known about the time course of the behavior matching, even though there is evidence that dyads synchronize oscillatory motions (e.g., postural sway). This study examined the temporal structure of nonoscillatory actions-language, facial, and gestural behaviors-produced during a route communication task. The focus was the temporal relationship between matching behaviors in the interlocutors (e.g., facial behavior in one interlocutor vs. the same facial behavior in the other interlocutor). Cross-recurrence analysis revealed that within each category tested (language, facial, gestural), interlocutors synchronized matching behaviors, at temporal lags short enough to provide imitation of one interlocutor by the other, from one conversational turn to the next. Both social and cognitive variables predicted the degree of temporal organization. These findings suggest that the temporal structure of matching behaviors provides low-level and low-cost resources for human interaction.

The linguistic and embodied nature of conceptual processing

Recent theories of cognition have argued that embodied experience is important for conceptual processing. Embodiment can be contrasted with linguistic factors such as the typical order in which words appear in language. Here, we report four experiments that investigated the conditions under which embodiment and linguistic factors determine performance. Participants made speeded judgments about whether pairs of words or pictures were semantically related or had an iconic relationship. The embodiment factor was operationalized as the degree to which stimulus pairs were presented in the spatial configurations in which they usually occur (i.e., an iconic configuration, e.g., attic presented above basement). The linguistic factor was operationalized as the frequency of the stimulus pairs in language. The embodiment factor predicted error rates and response time better for pictures, whereas the linguistic factor predicted error rates and response time better for words. These findings were modified by task, with the embodiment factor being strongest in iconicity judgments for pictures and the linguistic factor being strongest in semantic judgments for words. Both factors predicted error rates and response time for both semantic and iconicity judgments. These findings support the view that conceptual processing is both linguistic and embodied, with a bias for the embodiment or the linguistic factor depending on the nature of the task and the stimuli.

COGNITIVELY INSPIRED NLP-BASED KNOWLEDGE REPRESENTATIONS: FURTHER EXPLORATIONS OF LATENT SEMANTIC ANALYSIS

Natural-language based knowledge representations borrow their expressiveness from the semantics of language. One such knowledge representation technique is Latent semantic analysis (LSA), a statistical, corpus-based method for representing knowledge. It has been successfully used in a variety of applications including intelligent tutoring systems, essay grading and coherence metrics. The advantage of LSA is that it is efficient in representing world knowledge without the need for manual coding of relations and that it has in fact been considered to simulate aspects of human knowledge representation. An overview of LSA applications will be given, followed by some further explorations of the use of LSA. These explorations focus on the idea that the power of LSA can be amplified by considering semantic fields of text units instead of pairs of text units. Examples are given for semantic networks, category membership, typicality, spatiality and temporality, showing new evidence for LSA as a mechanism for knowledge representation. The results of such tests show that while the mechanism behind LSA is unique, it is flexible enough to replicate results in different corpora and languages.

Measuring Aboutness of an Entity in a Text

In many information retrieval and selection tasks it is valuable to score how much a text is about a certain entity and to compute how much the text discusses the entity with respect to a certain viewpoint. In this paper we are interested in giving an aboutness score to a text, when the input query is a person name and we want to measure the aboutness with respect to the biographical data of that person. We present a graph-based algorithm and compare its results with other approaches.

The role of discourse structure and response time in multimodal communication

In an ongoing project on multimodal communication in humans and agents [1], we investigate the interaction between two linguistic modalities (prosody and dialog structure) and two non-linguistic modalities (eye gaze and facial expressions). The goal is to gain a better understanding of the use of communicative channels in discourse and can subsequently aid the development of more effective animated conversational agents. We studied conversations between humans involved in the Map Task scenario whereby an Instruction Giver (IG) navigates an Instruction Follower (IF) from a starting point to an end point on a map [2].