Milene Bonte

Who Is Saying "What"? Brain-Based Decoding of Human Voice and Speech

Can we decipher speech content (“what” is being said) and speaker identity (“who” is saying it) from observations of brain activity of a listener? Here, we combine functional magnetic resonance imaging with a data-mining algorithm and retrieve what and whom a person is listening to from the neural fingerprints that speech and voice signals elicit in the listeners auditory cortex. These cortical fingerprints are spatially distributed and insensitive to acoustic variations of the input so as to permit the brain-based recognition of learned speech from unknown speakers and of learned voices from previously unheard utterances. Our findings unravel the detailed cortical layout and computational properties of the neural populations at the basis of human speech recognition and speaker identification.

The long road to automation: Neurocognitive development of letter-speech sound processing

In transparent alphabetic languages, the expected standard for complete acquisition of letter–speech sound associations is within one year of reading instruction. The neural mechanisms underlying the acquisition of letter–speech sound associations have, however, hardly been investigated. The present article describes an ERP study with beginner and advanced readers in which the influence of letters on speech sound processing is investigated by comparing the MMN to speech sounds presented in isolation with the MMN to speech sounds accompanied by letters. Furthermore, SOA between letter and speech sound presentation was manipulated in order to investigate the development of the temporal window of integration for letter–speech sound processing. Beginner readers, despite one year of reading instruction, showed no early letter–speech sound integration, that is, no influence of the letter on the evocation of the MMN to the speech sound. Only later in the difference wave, at 650 msec, was an influence of the letter on speech sound processing revealed. Advanced readers, with 4 years of reading instruction, showed early and automatic letter–speech sound processing as revealed by an enhancement of the MMN amplitude, however, at a different temporal window of integration in comparison with experienced adult readers. The present results indicate a transition from mere association in beginner readers to more automatic, but still not “adult-like,” integration in advanced readers. In contrast to general assumptions, the present study provides evidence for an extended development of letter–speech sound integration.

Fighting food temptations: The modulating effects of short-term cognitive reappraisal, suppression and up-regulation on mesocorticolimbic activity related to appetitive motivation

The premise of cognitive therapy is that one can overcome the irresistible temptation of highly palatable foods by actively restructuring the way one thinks about food. Testing this idea, participants in the present study were instructed to passively view foods, up-regulate food palatability thoughts, apply cognitive reappraisal (e.g., thinking about health consequences), or suppress food palatability thoughts and cravings. We examined whether these strategies affect self-reported food craving and mesocorticolimbic activity as assessed by functional magnetic resonance imaging. It was hypothesized that cognitive reappraisal would most effectively inhibit the mesocorticolimbic activity and associated food craving as compared to suppression. In addition, it was hypothesized that suppression would lead to more prefrontal cortex activity, reflecting the use of more control resources, as compared to cognitive reappraisal. Self-report results indicated that up-regulation increased food craving compared to the other two conditions, but that there was no difference in craving between the suppression and cognitive reappraisal strategy. Corroborating self-report results, the neuroimaging results showed that up-regulation increased activity in important regions of the mesocorticolimbic circuitry, including the ventral tegmental area, ventral striatum, operculum, posterior insular gyrus, medial orbitofrontal cortex and ventromedial prefrontal cortex. Contrary to our hypothesis, suppression more effectively decreased activity in the core of the mesocorticolimbic circuitry (i.e., ventral tegmental area and ventral striatum) compared to cognitive reappraisal. Overall, the results support the contention that appetitive motivation can be modulated by the application of short-term cognitive control strategies.

Cross-modal enhancement of the MMN to speech-sounds indicates early and automatic integration of letters and speech-sounds

Recently brain imaging evidence indicated that letter/speech-sound integration, necessary for establishing fluent reading, takes place in auditory association areas and that the integration is influenced by stimulus onset asynchrony (SOA) between the letter and the speech-sound. In the present study, we used a specific ERP measure known for its automatic character, the mismatch negativity (MMN), to investigate the time course and automaticity of letter/speech-sound integration. We studied the effect of visual letters and SOA on the MMN elicited by a deviant speech-sound. We found a clear enhancement of the MMN by simultaneously presenting a letter, but without changing the auditory stimulation. This enhancement diminishes linearly with increasing SOA. These results suggest that letters and speech-sounds are processed as compound stimuli early and automatically in the auditory association cortex of fluent readers and that this processing is strongly dependent on timing.

Auditory cortical tuning to statistical regularities in phonology

OBJECTIVE Ample behavioral evidence suggests that distributional properties of the language environment influence the processing of speech. Yet, how these characteristics are reflected in neural processes remains largely unknown. The present ERP study investigates neurophysiological correlates of phonotactic probability: the distributional frequency of phoneme combinations. METHODS We employed an ERP measure indicative of experience-dependent auditory memory traces, the mismatch negativity (MMN). We presented pairs of non-words that differed by the degree of phonotactic probability in a modified passive oddball design that minimizes the contribution of acoustic processes. RESULTS In Experiment 1 the non-word with high phonotactic probability (notsel) elicited a significantly enhanced MMN as compared to the non-word with low phonotactic probability (notkel). In Experiment 2 this finding was replicated with a non-word pair with a smaller acoustic difference (notsel-notfel). An MMN enhancement was not observed in a third acoustic control experiment with stimuli having comparable phonotactic probability (so-fo). CONCLUSIONS Our data suggest that auditory cortical responses to phoneme clusters are modulated by statistical regularities of phoneme combinations. SIGNIFICANCE This study indicates that the language environment is relevant in shaping the neural processing of speech. Furthermore, it provides a potentially useful design for investigating implicit phonological processing in children with anomalous language functions like dyslexia.

Deviant neurophysiological responses to phonological regularities in speech in dyslexic children

Developmental dyslexia is strongly associated with a phonological deficit. Yet, implicit phonological processing (in)capacities in dyslexia remain relatively unexplored. Here we use a neurophysiological response sensitive to experience-dependent auditory memory traces, the mismatch negativity (MMN), to investigate implicit phonological processing of natural speech in dyslexic and normally reading children. In a modified passive oddball design that minimizes the contribution of acoustic processes, we presented non-words that differed by the degree of phonotactic probability, i.e. the distributional frequency of phoneme combinations in a given language. Overall morphology of ERP responses to the non-words indicated comparable processing of acoustic-phonetic stimulus differences in both children groups. Consistent with previous findings in adults, normally reading children showed a significantly stronger MMN response to the non-word with high phonotactic probability (notsel) as compared to the non-word with low phonotactic probability (notkel), suggesting auditory cortical tuning to statistical regularities of phoneme combinations. In contrast, dyslexic children did not show this sensitivity to phonotactic probability. These findings indicate that the phonological problems often reported in dyslexia relate to a subtle deficit in the implicit phonetic-phonological processing of natural speech.

Hearing illusory sounds in noise: the timing of sensory-perceptual transformations in auditory cortex.

Constructive mechanisms in the auditory system may restore a fragmented sound when a gap in this sound is rendered inaudible by noise to yield a continuity illusion. Using combined psychoacoustic and electroencephalography experiments in humans, we found that the sensory-perceptual mechanisms that enable restoration suppress auditory cortical encoding of gaps in interrupted sounds. When physically interrupted tones are perceptually restored, stimulus-evoked synchronization of cortical oscillations at approximately 4 Hz is suppressed as if physically uninterrupted sounds were encoded. The restoration-specific suppression is induced most strongly in primary-like regions in the right auditory cortex during illusorily filled gaps and also shortly before and after these gaps. Our results reveal that spontaneous modulations in slow evoked auditory cortical oscillations that are involved in encoding acoustic boundaries may determine the perceived continuity of sounds in noise. Such fluctuations could facilitate stable hearing of fragmented sounds in natural environments.

Task-Dependent Decoding of Speaker and Vowel Identity from Auditory Cortical Response Patterns

Selective attention to relevant sound properties is essential for everyday listening situations. It enables the formation of different perceptual representations of the same acoustic input and is at the basis of flexible and goal-dependent behavior. Here, we investigated the role of the human auditory cortex in forming behavior-dependent representations of sounds. We used single-trial fMRI and analyzed cortical responses collected while subjects listened to the same speech sounds (vowels /a/, /i/, and /u/) spoken by different speakers (boy, girl, male) and performed a delayed-match-to-sample task on either speech sound or speaker identity. Univariate analyses showed a task-specific activation increase in the right superior temporal gyrus/sulcus (STG/STS) during speaker categorization and in the right posterior temporal cortex during vowel categorization. Beyond regional differences in activation levels, multivariate classification of single trial responses demonstrated that the success with which single speakers and vowels can be decoded from auditory cortical activation patterns depends on task demands and subjects behavioral performance. Speaker/vowel classification relied on distinct but overlapping regions across the (right) mid-anterior STG/STS (speakers) and bilateral mid-posterior STG/STS (vowels), as well as the superior temporal plane including Heschls gyrus/sulcus. The task dependency of speaker/vowel classification demonstrates that the informative fMRI response patterns reflect the top-down enhancement of behaviorally relevant sound representations. Furthermore, our findings suggest that successful selection, processing, and retention of task-relevant sound properties relies on the joint encoding of information across early and higher-order regions of the auditory cortex.

Developmental changes in ERP correlates of spoken word recognition during early school years: A phonological priming study.

OBJECTIVE We investigated event-related potential (ERP) correlates of developmental changes in spoken word recognition during early school years. We focused on implicit processing of word onsets as this may change considerably due to vocabulary growth and reading acquisition. METHODS Subjects were pre-schoolers (5-6 years), beginning readers (7-8 years) and adults. Two experiments examined phonological onset priming effects on ERP measures in an auditory lexical decision task. Primes were words (Expt. 1) or non-words (Expt. 2). RESULTS ERPs elicited by words showed clear developmental changes in ERP latency and morphology, especially with regard to early negativities like the N1. Prominent priming effects were an enhanced N400 amplitude due to alliterating word primes in beginning readers, and typical phonological N400 reductions due to alliterating non-word primes in all groups. Priming further led to opposite early effects with word primes (N1/N(200) reduction) vs. non-word primes (P1/N1 enhancement) in all groups. CONCLUSIONS These changes in ERP morphology and priming effects suggest that the lexical system undergoes substantial restructuring at the level of phonological processing and representation. In particular, our results indicate distinct processing of word onsets in beginning readers, confirming that vocabulary growth and the acquisition of reading may critically contribute to the formation of a fully segmental lexical system.

Brain-Based Translation: fMRI Decoding of Spoken Words in Bilinguals Reveals Language-Independent Semantic Representations in Anterior Temporal Lobe

Bilinguals derive the same semantic concepts from equivalent, but acoustically different, words in their first and second languages. The neural mechanisms underlying the representation of language-independent concepts in the brain remain unclear. Here, we measured fMRI in human bilingual listeners and reveal that response patterns to individual spoken nouns in one language (e.g., “horse” in English) accurately predict the response patterns to equivalent nouns in the other language (e.g., “paard” in Dutch). Stimuli were four monosyllabic words in both languages, all from the category of “animal” nouns. For each word, pronunciations from three different speakers were included, allowing the investigation of speaker-independent representations of individual words. We used multivariate classifiers and a searchlight method to map the informative fMRI response patterns that enable decoding spoken words within languages (within-language discrimination) and across languages (across-language generalization). Response patterns discriminative of spoken words within language were distributed in multiple cortical regions, reflecting the complexity of the neural networks recruited during speech and language processing. Response patterns discriminative of spoken words across language were limited to localized clusters in the left anterior temporal lobe, the left angular gyrus and the posterior bank of the left postcentral gyrus, the right posterior superior temporal sulcus/superior temporal gyrus, the right medial anterior temporal lobe, the right anterior insula, and bilateral occipital cortex. These results corroborate the existence of “hub” regions organizing semantic-conceptual knowledge in abstract form at the fine-grained level of within semantic category discriminations.