Emmanuel Ponsot

Cracking the social code of speech prosody using reverse correlation

Significance In speech, social evaluations of a speaker’s dominance or trustworthiness are conveyed by distinguishing, but little-understood, pitch variations. This work describes how to combine state-of-the-art vocal pitch transformations with the psychophysical technique of reverse correlation and uses this methodology to uncover the prosodic prototypes that govern such social judgments in speech. This finding is of great significance, because the exact shape of these prototypes, and how they vary with sex, age, and culture, is virtually unknown, and because prototypes derived with the method can then be reapplied to arbitrary spoken utterances, thus providing a principled way to modulate personality impressions in speech. Human listeners excel at forming high-level social representations about each other, even from the briefest of utterances. In particular, pitch is widely recognized as the auditory dimension that conveys most of the information about a speaker’s traits, emotional states, and attitudes. While past research has primarily looked at the influence of mean pitch, almost nothing is known about how intonation patterns, i.e., finely tuned pitch trajectories around the mean, may determine social judgments in speech. Here, we introduce an experimental paradigm that combines state-of-the-art voice transformation algorithms with psychophysical reverse correlation and show that two of the most important dimensions of social judgments, a speaker’s perceived dominance and trustworthiness, are driven by robust and distinguishing pitch trajectories in short utterances like the word “Hello,” which remained remarkably stable whether male or female listeners judged male or female speakers. These findings reveal a unique communicative adaptation that enables listeners to infer social traits regardless of speakers’ physical characteristics, such as sex and mean pitch. By characterizing how any given individual’s mental representations may differ from this generic code, the method introduced here opens avenues to explore dysprosody and social-cognitive deficits in disorders like autism spectrum and schizophrenia. In addition, once derived experimentally, these prototypes can be applied to novel utterances, thus providing a principled way to modulate personality impressions in arbitrary speech signals.

Are Rising Sounds Always Louder? Influences of Spectral Structure and Intensity-Region on Loudness Sensitivity to Intensity-Change Direction

In a previous study, a robust asymmetry in global loudness was observed between rising and falling-intensity 1-kHz tones, pointing out the existence of a mechanism specifically sensitive to sound intensity direction [Ponsot et al., Attention Perception, & Psychophysics, 77(3), 907-920 (2015)]. The properties of this “direction-sensitive” mechanism are further explored in the present study, where its dependence on two stimulus characteristics, the spectral content and the intensity region is examined. In a first experiment, the global loudness of rising and falling-intensity sounds varying over 15dB ranges was assessed in a magnitude estimation task. The ramps had various spectral contents (pure tones from 250 Hz to 8 kHz and broadband noises) and were presented in different intensity-regions (from [50-65 dB SPL] to [70-85 dB SPL]). Significant asymmetries were observed across the different frequencies but not for broadband noises. In addition, a significant interaction between the direction of intensity-change and the region of intensity was observed for tones and noises. This latter effect was specifically addressed in a second experiment using an adaptive loudness-matching procedure, in which asymmetries were inferred from pairwise comparisons, both for 1-kHz tones and for white noises presented in mid and high intensity-regions. As in Exp. 1, greater asymmetries were observed for tones compared to noises; however significant asymmetries were found for noises as well. Furthermore, for both tones and noises, the size of the asymmetries was significantly decreased with the intensity-region when the sound pairs were composed of a falling followed by a rising stimulus. These results are discussed in the light of recent physiological and neuroscience studies showing that spectrally structured looming sounds are treated specially by the brain.

Uncovering mental representations of smiled speech using reverse correlation

Which spectral cues underlie the perceptual processing of smiles in speech? Here, the question was addressed using reverse-correlation in the case of the isolated vowel [a]. Listeners were presented with hundreds of pairs of utterances with randomly manipulated spectral characteristics and were asked to indicate, in each pair, which was the most smiling. The analyses revealed that they relied on robust spectral representations that specifically encoded vowels formants. These findings demonstrate the causal role played by formants in the perception of smile. Overall, this paper suggests a general method to estimate the spectral bases of high-level (e.g., emotional/social/paralinguistic) speech representations.

Effect of sound duration on loudness estimates of increasing and decreasing intensity sounds

The influence of sound duration on global loudness of non-stationary stimuli was investigated. Loudness of 2 and 6-s increasing and decreasing intensity sounds with different ranges of intensity-variation was assessed using a magnitude estimation procedure. Results once again uphold the existence of a loudness difference between the two patterns: while they only differ in their temporal profile, increasing sounds were perceived louder than decreasing sounds. In addition, global loudness estimates were increased with duration for the two types of sounds, and a small but significant interaction occurred between type and duration. A contrast analysis revealed that while global loudness of increasing and decreasing sounds raised with duration in a similar way in the case of low and moderate intensities (below 75 dB SPL), global loudness was significantly more affected by duration with increasing than with decreasing intensity profiles for high-intensity stimuli. This result suggests the existence of an underl...

Reply to Knight et al.: The complexity of inferences from speech prosody should be addressed using data-driven approaches

We are glad our proposed methodological approach (1) raises interest in the community. Knight et al. (2) make two important theoretical considerations that we would like to further develop here. The first point they raise concerns the specificity of the pitch prototype of dominance/trustworthiness: They argue one should demonstrate that these prototypes are specific (i.e., not shared by other emotional or linguistic traits). Dominance and trustworthiness, while difficult to explicitly define, are not vague concepts: As is the case for social faces in vision (3), they constitute the two principal dimensions of the social space experimentally derived for speech with two-syllable utterances (4). These traits are therefore robust … [↵][1]1To whom correspondence should be addressed. Email: emmanuel.ponsot{at}ens.fr. [1]: #xref-corresp-1-1

CLEESE: An open-source audio-transformation toolbox for data-driven experiments in speech and music cognition

Over the past few years, the field of visual social cognition and face processing has been dramatically impacted by a series of data-driven studies employing computer-graphics tools to synthesize arbitrary meaningful facial expressions. In the auditory modality, reverse correlation is traditionally used to characterize sensory processing at the level of spectral or spectro-temporal stimulus properties, but not higher-level cognitive processing of e.g. words, sentences or music, by lack of tools able to manipulate the stimulus dimensions that are relevant for these processes. Here, we present an open-source audio-transformation toolbox, called CLEESE, able to systematically randomize the prosody/melody of existing speech and music recordings. CLEESE works by cutting recordings in small successive time segments (e.g. every successive 100 milliseconds in a spoken utterance), and applying a random parametric transformation of each segment’s pitch, duration or amplitude, using a new Python-language implementation of the phase-vocoder digital audio technique. We present here two applications of the tool to generate stimuli for studying intonation processing of interrogative vs declarative speech, and rhythm processing of sung melodies.

A methodological framework to derive the mental code of emotional and social inferences in sound

Most research carried out so far in auditory emotion has been based on theory-driven experiments. Such experiments, which involve a limited number of conditions, help in refining existing models or theories but cannot expose elements or processes that were not expected and targeted initially. Here, I will present a methodological framework based on a data-driven approach that allows to probe the mechanisms of auditory cognition in a space that is not constrained by a priori hypotheses. This framework uses signal-processing algorithms for manipulating the acoustical characteristics of an input sound and create a huge number of parametrically-manipulated, natural expressive variations of this sound, which are then used as stimuli in psychophysical experiments employing a reverse-correlation technique. I will present different contexts in which this framework has been used, in particular to explore the processing of speech prosodic dimensions in the formation of social impressions. Because this approach offers a principled way to reverse-engineer any high-level judgment in any individual, it should be helpful to understand the algorithms that the brain uses to build high-level emotional or social impressions from the acoustical characteristics of sound in their complexity.Most research carried out so far in auditory emotion has been based on theory-driven experiments. Such experiments, which involve a limited number of conditions, help in refining existing models or theories but cannot expose elements or processes that were not expected and targeted initially. Here, I will present a methodological framework based on a data-driven approach that allows to probe the mechanisms of auditory cognition in a space that is not constrained by a priori hypotheses. This framework uses signal-processing algorithms for manipulating the acoustical characteristics of an input sound and create a huge number of parametrically-manipulated, natural expressive variations of this sound, which are then used as stimuli in psychophysical experiments employing a reverse-correlation technique. I will present different contexts in which this framework has been used, in particular to explore the processing of speech prosodic dimensions in the formation of social impressions. Because this approach offe...

Effect of Pitch on the Asymmetry in Global Loudness Between Rising- and Falling-Intensity Sounds

The global loudness of a varying intensity sound is greater when the intensity increases than when it decreases. This global loudness asymmetry was found to be larger for pure tones than for broadband noises. In this study, our aim was to determine whether this difference between pure tones and noises is due to the difference in bandwidth between sounds or to the difference in the strength of the sensation of pitch. The loudness asymmetry was measured for broadband and for narrow-band signals that do or do not elicit a sensation of pitch. The asymmetry was greater for sounds that elicit a sensation of pitch whatever their bandwidth. The loudness model for time varying sounds [1] predicted well the asymmetry for the broadband noise that does not elicit a sensation of pitch and for a multi-tonal sound. For the other sounds the asymmetry was greater than predicted. It is known that loudness and pitch interact. The difference in asymmetry between sounds that elicit pitch and sounds that do not elicit pitch might be due to this interaction.

Some factors influencing loudness asymmetries between rising and falling-intensity stimuli

Previous research demonstrated that the loudness asymmetry between 1-kHz rising and falling-intensity tones is a robust phenomenon, whose origins still remain unclear. In the present study, this phenomenon was further examined as a function of two stimuli characteristics: the spectral content and the intensity-region. In a first experiment, the global loudness of rising and falling-intensity sounds with various spectral contents (pure tones from 250 Hz to 8 kHz and broadband noises) presented in different intensity-regions (from [50–65 dB SPL] to [70–85 dB SPL]) was assessed in an absolute magnitude estimation task. Significant asymmetries were found for tones at all frequencies, but not for broadband noises. In addition, a significant interaction between the stimulus direction and the intensity-region was observed for both tones and noises. This interaction was further examined in a second experiment using an adaptive loudness-matching procedure. Although greater asymmetries were again observed for tones...

Erratum to: A robust asymmetry in loudness between rising- and falling-intensity tones

One sentence in the article’s Results section is not an accurate account of the work it refers to (Hellström, 1979, 2003). This sentence appears under the subheading Time order errors in the middle of the text. The amended formulation appears below. “One of the suggestions proposed to explain TOEs is that the two stimuli are weighted differently, a greater weight often being accorded to the second stimulus (Hellström 1979, 2003). As a consequence, when the levels of the two tones are in low intensity regions, the first stimulus is overestimated relative to the second (a positive TOE), and when they are in high regions, the first stimulus is underestimated relative to the second (a negative TOE).”