Rivka Levitan

Measuring acoustic-prosodic entrainment with respect to multiple levels and dimensions

In conversation, speakers become more like each other in various dimensions. This phenomenon, commonly called entrainment, coordination, or alignment, is widely believed to be crucial to the success and naturalness of human interactions. We investigate entrainment in four acoustic and prosodic dimensions. We explore whether speakers coordinate with each other in these dimensions over the conversation as a whole as well as on a turn-by-turn basis and in both relative and absolute terms, and whether this coordination improves over the course of the conversation.

Entrainment in Speech Preceding Backchannels.

In conversation, when speech is followed by a backchannel, evidence of continued engagement by ones dialogue partner, that speech displays a combination of cues that appear to signal to ones interlocutor that a backchannel is appropriate. We term these cues back-channel-preceding cues (BPC)s, and examine the Columbia Games Corpus for evidence of entrainment on such cues. Entrainment, the phenomenon of dialogue partners becoming more similar to each other, is widely believed to be crucial to conversation quality and success. Our results show that speaking partners entrain on BPCs; that is, they tend to use similar sets of BPCs; this similarity increases over the course of a dialogue; and this similarity is associated with measures of dialogue coordination and task success.

Cross-Language Prominence Detection

We explore the ability to perform automatic prosodic analysis in one language using models trained on another. If we are successful, we should be able to identify prosodic elements in a language for which little or no prosodically labeled training data is available, using models trained on a language for which such training data exists. Given the laborious nature of manual prosodic annotation, such a process would vastly improve our ability to identify prosodic events in many languages and therefore to make use of such information in downstream processing tasks. The task we address here is the detection of intonational prominence, performing experiments using material from four languages: American English, Italian, French and German. While we do find that cross-language prominence detection is possible, we also find significant language-dependent differences. While we hypothesized that language family might serve as a reliable predictor of cross-language prosodic event detection accuracy, in our experiments this did not prove to be the case. Based upon our results, we suggest some directions that may be able to improve our cross-language approach.

Three ToBI-based measures of prosodic entrainment and their correlations with speaker engagement

Entrainment is the propensity of conversational partners to align different aspects of their communicative behavior. In this study we present three novel measures of prosodic entrainment based on intonational contours as defined by the ToBI conventions for prosodic description. Each of these measures estimates the similarity of contours used by speakers in different ways: by means of the perplexity of n-gram models, the Levenshtein distance, and the Kullback-Leibler divergence measure. We report significant correlations between each of these measures and manual annotations of a number of social variables related to the level of engagement of speakers, in a corpus of task-oriented dialogues in Standard American English.

Entrainment, dominance and alliance in supreme court hearings

A major goal of the Cognitive Infocommunication approach is to develop applications in which human and artificial cognitive systems are made to work more effectively. A critical step in this process is improving our understanding of human–human interaction so that it may be modeled more closely. Our work addresses this task by examining the role of entrainment – the propensity of conversational partners to behave like one another – in (1) the production of conversational fillers (CFs) and acoustic intensity; (2) patterns of turn-taking; and (3) Linguistic Style. Markers and how all of these relate to power relations, conflict, and voting behavior in a corpus of speech produced by justices and lawyers during oral arguments of the U.S. Supreme Court in the 2001 term. We examine several different measures of entrainment in justice–lawyer pairs to see whether or not they are related to justices’ favorable or unfavorable votes for the lawyer’s side. While two measures (a naive measure of similarity in CF rates and global similarity in CF phonetic realizations for the entire session) show no relationship, a third, which measures local entrainment in CFs in lawyer-justice pairs, does in fact identify a significant positive relationship between entrainment and justice votes. With respect to local entrainment in intensity, we found that lawyers do entrain more to justices than justices to lawyers, although there is no greater entrainment of female lawyers than of male lawyers. When we examine the relationship between entrainment in intensity and judicial voting, we find that, when justices voted for the petitioners, there is significant evidence of entrainment by both petitioners and respondents to justices. With respect to turn-taking behavior, we find that certain patterns of overlaps in turn exchanges between justices and lawyers are correlated with justices’ voting behavior for four of the justices in our corpus. Finally, we find that there are lexical cues to divisiveness within the Court itself that can distinguish cases with close verdicts from cases with unanimous verdicts. We link these results to the possibility of building cognitive info-communication interfaces that exploit features of human–human entrainment for increasing effectiveness of human–machine interactions.

Acoustic-prosodic entrainment in Slovak, Spanish, English and Chinese: A cross-linguistic comparison

It is well established that speakers of Standard American English entrain, or become more similar to each other as they speak, in acoustic-prosodic features of their speech as well as other behaviors. Entrainment in other languages is less well understood. This work uses a variety of metrics to measure acoustic-prosodic entrainment in four comparable corpora of task-oriented conversational speech in Slovak, Spanish, English and Chinese. We report the results of these experiments and describe trends and patterns that can be observed from comparing acoustic-prosodic entrainment in these four languages. We find evidence of a variety of forms of entrainment across all the languages studied, with some evidence of individual differences as well within the languages.

Detecting Retries of Voice Search Queries

When a system fails to correctly recognize a voice search query, the user will frequently retry the query, either by repeating it exactly or rephrasing it in an attempt to adapt to the system’s failure. It is desirable to be able to identify queries as retries both offline, as a valuable quality signal, and online, as contextual information that can aid recognition. We present a method than can identify retries offline with 81% accuracy using similarity measures between two subsequent queries as well as system and user signals of recognition accuracy. The retry rate predicted by this method correlates significantly with a gold standard measure of accuracy, suggesting that it may be useful as an offline predictor of accuracy.

Implementing Acoustic-Prosodic Entrainment in a Conversational Avatar.

Entrainment, aka accommodation or alignment, is the phenomenon by which conversational partners become more similar to each other in behavior. While there has been much work on some behaviors there has been little on entrainment in speech and even less on how Spoken Dialogue Systems (SDS) which entrain to their users’ speech can be created. We present an architecture and algorithm for implementing acoustic-prosodic entrainment in SDS and show that speech produced under this algorithm conforms to the feature targets, satisfying the properties of entrainment behavior observed in human-human conversations. We present results of an extrinsic evaluation of this method, comparing whether subjects are more likely to ask advice from a conversational avatar that entrains vs. one that does not, in English, Spanish and Slovak SDS.

Combining Acoustic-Prosodic, Lexical, and Phonotactic Features for Automatic Deception Detection.

Improving methods of automatic deception detection is an important goal of many researchers from a variety of disciplines, including psychology, computational linguistics, and criminology. We present a system to automatically identify deceptive utterances using acoustic-prosodic, lexical, syntactic, and phonotactic features. We train and test our system on the Interspeech 2016 ComParE challenge corpus, and find that our combined features result in performance well above the challenge baseline on the development data. We also perform feature ranking experiments to evaluate the usefulness of each of our feature sets. Finally, we conduct a cross-corpus evaluation by training on another deception corpus and testing on the ComParE corpus.

Automatically Classifying Self-Rated Personality Scores from Speech.

Automatic personality recognition is useful for many computational applications, including recommendation systems, dating websites, and adaptive dialogue systems. There have been numerous successful approaches to classify the “Big Five” personality traits from a speaker’s utterance, but these have largely relied on judgments of personality obtained from external raters listening to the utterances in isolation. This work instead classifies personality traits based on self-reported personality tests, which are more valid and more difficult to identify. Our approach, which uses lexical and acoustic-prosodic features, yields predictions that are between 6.4% and 19.2% more accurate than chance. This approach predicts Opennessto-Experience and Neuroticism most successfully, with less accurate recognition of Extroversion. We compare the performance of classification and regression techniques, and also explore predicting personality clusters.