Maciej Karpiński

Gesture, prosody and lexicon in task-oriented dialogues: multimedia corpus recording and labelling

The aim of the DiaGest Project is to study interdependencies between gesture, lexicon, and prosody in Polish dialogues. The material under study comprises three tasks realised by twenty pairs of subjects. Two tasks involve instructional, task-oriented dialogues, while the third is based on a question answering procedure. A system for corpus labelling is currently being designed on the basis of current standards. The corpus will be annotated for gestures, lexical content of utterances, intonation and rhythm. In order to relate various phenomena to the contextualized meaning of dialogue utterances, the material will also be tagged in terms of dialogue acts. Synchronised tags will be placed in respective annotation tiers in ELAN. A number of detailed studies related to the problems of gesture-prosody, gesture-lexicon and prosody-lexicon interactions will be carried out on the basis of the tagged material.

The Boundaries of Language: Dealing with Paralinguistic Features

ABSTRACT Maciej Karpiński. The Boundaries of Language: Dealing with Paralinguistic Features. Lingua Posnaniensis, vol. LIV (2)/2012. The Poznań Society for the Advancement of the Arts and Sciences. PL ISSN 0079-4740, ISBN 978-83-7654-252-2, pp. 37-54. The paralinguistic component of communication attracted a great deal of attention from contemporary linguists in the 1960s. The seminal works written then by Trager, Crystal and others had a powerful influence on the concept of paralanguage that lasted for many years. But, with the focus shifting towards the socio-psychological context of communication in the 1970s, the development of spoken corpora and databases and the significant progress in speech technology in the 1980s and 1990s, the need has arisen for a more comprehensive, coherent and formalised - but also flexible - approach to paralinguistic features. This study advances some preliminary proposals for a revised treatment of paralanguage that would meet some of these requirements and provide a conceptual basis for a new system of annotation for paralinguistic features. A range of views on paralinguistic features, which come mostly from the fields of speech prosody and gesture analysis, are briefly discussed. A number of assumptions and postulates are formulated to allow for a more consistent approach to paralinguistic features. The study suggests that there should be more reliance on continua than on binary categorisations of features, that multi-functionality and multimodality should be fully acknowledged and that clear distinctions should be made among the levels of description, and between the properties of speakers and the speech signal itself.

From Speech and Gestures to Dialogue Acts

In the present paper, selected problems related to the annotation of the DiaGest corpus are discussed. A system of dialogue acts is proposed along with a conceptual framework that allows for independent labelling of the contributions provided by various modalities and channels. Both auditory and visual modalities are considered. Four channels are defined as major ways of providing quasi-independent modal contributions. A four-dimensional categorisation of dialogue acts is proposed. It includes a separate dimension for attitude-related tags. Dialogue acts are conceptualised as multidimensional entities built on the basis of modal contributions provided by respective channels.

Pitch Syntax Violations Are Linked to Greater Skin Conductance Changes, Relative to Timbral Violations – The Predictive Role of the Reward System in Perspective of Cortico–subcortical Loops

According to contemporary opinion emotional reactions to syntactic violations are due to surprise as a result of the general mechanism of prediction. The classic view is that, the processing of musical syntax can be explained by activity of the cerebral cortex. However, some recent studies have indicated that subcortical brain structures, including those related to the processing of emotions, are also important during the processing of syntax. In order to check whether emotional reactions play a role in the processing of pitch syntax or are only the result of the general mechanism of prediction, the comparison of skin conductance levels reacting to three types of melodies were recorded. In this study, 28 subjects listened to three types of short melodies prepared in Musical Instrument Digital Interface Standard files (MIDI) – tonally correct, tonally violated (with one out-of-key – i.e., of high information content), and tonally correct but with one note played in a different timbre. The BioSemi ActiveTwo with two passive Nihon Kohden electrodes was used. Skin conductance levels were positively correlated with the presented stimuli (timbral changes and tonal violations). Although changes in skin conductance levels were also observed in response to the change in timbre, the reactions to tonal violations were significantly stronger. Therefore, despite the fact that timbral change is at least as equally unexpected as an out-of-key note, the processing of pitch syntax mainly generates increased activation of the sympathetic part of the autonomic nervous system. These results suggest that the cortico–subcortical loops (especially the anterior cingulate – limbic loop) may play an important role in the processing of musical syntax.

The intonational realization of requests in polish task-oriented dialogues

In the present paper, the intonational realization of Request Action dialogue acts in Polish map task dialogues is analyzed. The study is focused on the Request External Action acts realized as single, well-formed intonational phrases. Basic pitch-related parameters are measured and discussed. Nuclear melodies are described and categorized, and some generalizations are formulated about their common realizations. Certain aspects of the grammatical form and phrase placement in the dialogue flow are also taken into account. The results will be employed in comparative studies and in the preparation of glottodidactic materials, but they may also prove useful in the field of speech synthesis or recognition.

Dimensions of intonation. Wiktor Jassem’s contribution to the studies on the melody of speech

Abstract Intonation is often considered the most problematic component of prosody. It may largely contribute to the meaning of an utterance and provide rich indexical information. The complexity of the intonationrelated phenomena, from the acoustic level up to the cognitive process of semiosis, resulted in a variety of conceptualisations and divergent theoretical approaches. In the present text, an attempt is made to sketch the late Wiktor Jassem’s contribution to intonation studies, starting from his early work on colloquial British English to the studies focused on the Polish language as well as his general perspective on prosody and on the methods of its exploration. Some of his unpublished contribution to research projects on various aspects of prosody is also acknowledged. The present overview is based both on available written materials (publications, references, reports, reviews) and the author’s personal communication with Wiktor Jassem and his collaborators.

New challenges in psycholinguistics: Interactivity and alignment in interpersonal communication

Abstract In the present text, some recent changes in the perspective taken by psycholinguists in the study of language and communication are discussed. T heir interests seem to gradually shift from the study of language processing as an isolated and independent phenomenon towards inclusion of more interactional factors being indispensable components of interpersonal communication and involved in the process of communicative alignment. Alignment is here understood as a complex phenomenon that goes beyond increasing similarity of mental representations and related communicative behaviour. It simultaneously occurs on many levels and in various modalities, including those traditionally excluded from language study. A s a consequence, it implies not only more flexibility in the study of interpersonal communication but it also means a shift in the psycholinguistic methodology and probably also in the widely accepted picture of language and its limits.

Jakościowa i ilościowa analiza słownictwa oraz jednostek quasi- i nieleksykalnych wypowiedzi w dialogach zadaniowych „origami”

In this text, an analysis of the lexical material obtained from the recordings of Polish task-oriented dialogues is presented. Some findings regarding quasi- and non-lexical units are also mentioned. The dialogue task involved the reconstruction of a figure made of paper. It was carried out in two settings: Mutual visibility and limited visibility. The analyses of the vocabulary are focused on finding its specific features and the differences between the structures of the vocabularies of instruction givers and instruction followers in the two settings. Frequency lists as well as most frequent bigrams extracted from the material are discussed in terms of their relatedness to the settings, the roles of the speakers and some other factors.

Preliminary Prosodic and Gestural Characteristics of Instructing Acts in Polish Task-Oriented Dialogues

In the present study, selected properties of multimodal instructing acts are discussed. Realisations of the instructing acts extracted from a corpus of task-oriented dialogues are analysed in terms of their syntactic structure, prosodic properties and accompanying gestures. The syntactic structures found in the material are similar to those found in earlier studies on map task dialogues. Deictic vocabulary is more frequent in gesture-supported instructions. The mean relative pitch range is similar to the values obtained for instructions in earlier studies and different from the values for syntactically similar questions. As opposite to verbally ill-formed instructions, the well-formed ones tend to contain at least one gestural stroke. It is shown that the relative range of pitch frequency is higher in the gesture-accompanied instructing acts. It is also noticed that prosody and gesture may play similar roles in utterances.