Adriano Vilela Barbosa

Quantifying time-varying coordination of multimodal speech signals using correlation map analysis

This paper demonstrates an algorithm for computing the instantaneous correlation coefficient between two signals. The algorithm is the computational engine for analyzing the time-varying coordination between signals, which is called correlation map analysis (CMA). Correlation is computed around any pair of points in the two input signals. Thus, coordination can be assessed across a continuous range of temporal offsets and be detected even when changing over time due to temporal fluctuations. The correlation algorithm has two major features: (i) it is structurally similar to a tunable filter, requiring only one parameter to set its cutoff frequency (and sensitivity), (ii) it can be applied either uni-directionally (computing correlation based only on previous samples) or bi-directionally (computing correlation based on both previous and future samples). Computing instantaneous correlation for a range of time offsets between two signals produces a 2D correlation map, in which correlation is characterized as a function of time and temporal offset. Graphic visualization of the correlation map provides rapid assessment of how correspondence patterns progress through time. The utility of the algorithm and of CMA are exemplified using the spatial and temporal coordination of various audible and visible components associated with linguistic performance.

Movement Coordination during Conversation

Behavioral coordination and synchrony contribute to a common biological mechanism that maintains communication, cooperation and bonding within many social species, such as primates and birds. Similarly, human language and social systems may also be attuned to coordination to facilitate communication and the formation of relationships. Gross similarities in movement patterns and convergence in the acoustic properties of speech have already been demonstrated between interacting individuals. In the present studies, we investigated how coordinated movements contribute to observers’ perception of affiliation (friends vs. strangers) between two conversing individuals. We used novel computational methods to quantify motor coordination and demonstrated that individuals familiar with each other coordinated their movements more frequently. Observers used coordination to judge affiliation between conversing pairs but only when the perceptual stimuli were restricted to head and face regions. These results suggest that observed movement coordination in humans might contribute to perceptual decisions based on availability of information to perceivers.

Measuring the relation between speech acoustics and 2D facial motion

Presents a quantitative analysis of the relation between speech acoustics and the 2D video signal of the facial motion that occurs simultaneously. 2D facial motion is acquired using an ordinary video camera: after digitizing a video sequence, a search algorithm is used for tracking markers painted on the speakers face. Facial motion is represented by the 2D marker trajectories; whereas line spectrum pairs (LSP) coefficients are used to parameterize the speech acoustics. LSP coefficients and the marker trajectories are then used to train time-invariant and time-varying linear models, as well as nonlinear (neural network) models. These models are used to evaluate to what extent 2D facial motion is determined from speech acoustics. The correlation coefficients between measured and estimated trajectories are as high as 0.95. This estimation of facial motion from speech acoustics indicates a way to integrate audio and visual signals for efficient audio-visual speech coding.

“Diagnosis by Behavioral Observation” Home-Videosomnography – A Rigorous Ethnographic Approach to Sleep of Children with Neurodevelopmental Conditions

Introduction: Advanced video technology is available for sleep-laboratories. However, low-cost equipment for screening in the home setting has not been identified and tested, nor has a methodology for analysis of video recordings been suggested. Methods: We investigated different combinations of hardware/software for home-videosomnography (HVS) and established a process for qualitative and quantitative analysis of HVS-recordings. A case vignette (HVS analysis for a 5.5-year-old girl with major insomnia and several co-morbidities) demonstrates how methodological considerations were addressed and how HVS added value to clinical assessment. Results: We suggest an “ideal set of hardware/software” that is reliable, affordable (∼

Articulatory coordination of two vocal tracts

500) and portable (=2.8 kg) to conduct non-invasive HVS, which allows time-lapse analyses. The equipment consists of a net-book, a camera with infrared optics, and a video capture device. (1) We present an HVS-analysis protocol consisting of three steps of analysis at varying replay speeds: (a) basic overview and classification at 16× normal speed; (b) second viewing and detailed descriptions at 4–8× normal speed, and (c) viewing, listening, and in-depth descriptions at real-time speed. (2) We also present a custom software program that facilitates video analysis and note-taking (Annotator©), and Optical Flow software that automatically quantifies movement for internal quality control of the HVS-recording. The case vignette demonstrates how the HVS-recordings revealed the dimension of insomnia caused by restless legs syndrome, and illustrated the cascade of symptoms, challenging behaviors, and resulting medications. Conclusion: The strategy of using HVS, although requiring validation and reliability testing, opens the floor for a new “observational sleep medicine,” which has been useful in describing discomfort-related behavioral movement patterns in patients with communication difficulties presenting with challenging/disruptive sleep/wake behaviors.

Temporal characteristization of auditory‐visual coupling in speech

Abstract In this paper, we quantify the time-varying coordination of articulator motion for two speakers engaged in two face-to-face speaking tasks: repetition of simultaneously produced words (and word-pairs) and conversation. Correlation map analysis is used to compute the time-varying, correlation between pairs of signals for all possible pairings of signal values. The resulting two-dimensional map of correlation values captures the fluctuations that occur naturally in the coordination within and between speakers, regardless of whether the behaviors are intentionally synchronized or merely entrained (as in conversation). This not only provides a more accurate picture of the correlation between signals, but also underlines the importance of treating fluctuations as potentially important components of biological behavior, rather than discounting them simply as noise.

The labial viseme reconsidered: Evidence from production and perception.

This work examines the coupling between the acoustic and visual com- ponents of speech as it evolves through time. Previous work has shown a consistent correspondence between face motion and spectral acous- tics, and between fundamental frequency (F0) and rigid body motion of the head [Yehia et al. (2002), JPHON, 30, 555-568]. Although these correspondences have been estimated both for sentences and for running speech, the analyses have not taken into account the tempo- ral structure of speech. As a result, the role of temporal organization in multimodal speech cannot be assessed. The current study is a first effort to correct this deficit. We have developed an algorithm, based on recursive correlation, that computes the correlation between measurement domains (e.g., head motion and F0) as a time-varying function. Using this method, regions of high or low correlation, or of rapid transition (e.g., from high to low), can be associated with visual and auditory events. This analysis of the time-varying cou- pling of multimodal events has implications for speech planning and synchronization between speaker and listener.

Quantifying the time‐varying coordination between performer and audience.

Previous studies have demonstrated that the labial stops /p,b,m/ may be impossible to discriminate visually, leading to their designation as a single viseme. This perceptual limitation has engendered the belief that there are no visible differences in the production of /p,b,m/, with consequences for research in machine recognition, where production differences below the level of the viseme have been largely ignored. Kinematic studies using high‐speed cine, however, have previously documented systematic differences in the production of labial consonants. This study examines the degree to which visual /p,b,m/ are discriminable in production and perception. Two experiments—one designed to measure kinematic orofacial movement using optical flow analysis and one designed to test perceiver discrimination of /p,b,m/—were used to establish the absence/presence of systematic visual differences in bilabial productions, and to replicate the previous perception findings. Results from the optical flow analysis indicat...

Quantitative analysis of multimodal speech data

This study examines the coordination that occurs ubiquitously during behavioral interaction (e.g., linguistic, social, musical). Coordination, however, does not imply strict synchronization. Musicians in a quartet, for example, will deviate slightly from the measured beat, alternately playing slightly ahead or behind. This paper showcases a new algorithm for computing the continuous, instantaneous correlation between two signals, at ANY temporal offset, resulting in a two‐dimensional mapping of correlation and temporal offset that makes it possible to visualize and assess the time‐varying nature of the coordination. The algorithm is demonstrated through analysis of the extraordinary performer‐audience coordination evoked by Freddie Mercury during the rock group, Queen‘s, Live Aid Concert at Wembley Stadium in 1985.

Optical flow analysis for measuring tongue-motion

This study presents techniques for quantitatively analyzing coordination and kinematics in multimodal speech using video, audio and electromagnetic articulography (EMA) data. Multimodal speech research has flourished due to recent improvements in technology, yet gesture detection/annotation strategies vary widely, leading to difficulty in generalizing across studies and in advancing this field of research. We describe how FlowAnalyzer software can be used to extract kinematic signals from basic video recordings; and we apply a technique, derived from speech kinematic research, to detect bodily gestures in these kinematic signals. We investigate whether kinematic characteristics of multimodal speech differ dependent on communicative context, and we find that these contexts can be distinguished quantitatively, suggesting a way to improve and standardize existing gesture identification/annotation strategy. We also discuss a method, Correlation Map Analysis (CMA), for quantifying the relationship between speech and bodily gesture kinematics over time. We describe potential applications of CMA to multimodal speech research, such as describing characteristics of speech-gesture coordination in different communicative contexts. The use of the techniques presented here can improve and advance multimodal speech and gesture research by applying quantitative methods in the detection and description of multimodal speech.