Christian E. Stilp

Cochlea-scaled entropy, not consonants, vowels, or time, best predicts speech intelligibility.

Speech sounds are traditionally divided into consonants and vowels. When only vowels or only consonants are replaced by noise, listeners are more accurate understanding sentences in which consonants are replaced but vowels remain. From such data, vowels have been suggested to be more important for understanding sentences; however, such conclusions are mitigated by the fact that replaced consonant segments were roughly one-third shorter than vowels. We report two experiments that demonstrate listener performance to be better predicted by simple psychoacoustic measures of cochlea-scaled spectral change across time. First, listeners identified sentences in which portions of consonants (C), vowels (V), CV transitions, or VC transitions were replaced by noise. Relative intelligibility was not well accounted for on the basis of Cs, Vs, or their transitions. In a second experiment, distinctions between Cs and Vs were abandoned. Instead, portions of sentences were replaced on the basis of cochlea-scaled spectral entropy (CSE). Sentence segments having relatively high, medium, or low entropy were replaced with noise. Intelligibility decreased linearly as the amount of replaced CSE increased. Duration of signal replaced and proportion of consonants/vowels replaced fail to account for listener data. CSE corresponds closely with the linguistic construct of sonority (or vowel-likeness) that is useful for describing phonological systematicity, especially syllable composition. Results challenge traditional distinctions between consonants and vowels. Speech intelligibility is better predicted by nonlinguistic sensory measures of uncertainty (potential information) than by orthodox physical acoustic measures or linguistic constructs.

Auditory color constancy: Calibration to reliable spectral properties across nonspeech context and targets

Brief experience with reliable spectral characteristics of a listening context can markedly alter perception of subsequent speech sounds, and parallels have been drawn between auditory compensation for listening context and visual color constancy. In order to better evaluate such an analogy, the generality of acoustic context effects for sounds with spectral-temporal compositions distinct from speech was investigated. Listeners identified nonspeech sounds—extensively edited samples produced by a French horn and a tenor saxophone—following either resynthesized speech or a short passage of music. Preceding contexts were “colored” by spectral envelope difference filters, which were created to emphasize differences between French horn and saxophone spectra. Listeners were more likely to report hearing a saxophone when the stimulus followed a context filtered to emphasize spectral characteristics of the French horn, and vice versa. Despite clear changes in apparent acoustic source, the auditory system calibrated to relatively predictable spectral characteristics of filtered context, differentially affecting perception of subsequent target nonspeech sounds. This calibration to listening context and relative indifference to acoustic sources operates much like visual color constancy, for which reliable properties of the spectrum of illumination are factored out of perception of color.

Rapid efficient coding of correlated complex acoustic properties

Natural sounds are complex, typically changing along multiple acoustic dimensions that covary in accord with physical laws governing sound-producing sources. We report that, after passive exposure to novel complex sounds, highly correlated features initially collapse onto a single perceptual dimension, capturing covariance at the expense of unitary stimulus dimensions. Discriminability of sounds respecting the correlation is maintained, but is temporarily lost for sounds orthogonal or oblique to experienced covariation. Following extended experience, perception of variance not captured by the correlation is restored, but weighted only in proportion to total experienced covariance. A Hebbian neural network model captures some aspects of listener performance; an anti-Hebbian model captures none; but, a principal components analysis model captures the full pattern of results. Predictions from the principal components analysis model also match evolving listener performance in two discrimination tasks absent passive listening. These demonstrations of adaptation to correlated attributes provide direct behavioral evidence for efficient coding.

Predicting contrast effects following reliable spectral properties in speech perceptiona)

Vowel perception is influenced by precursor sounds that are resynthesized to shift frequency regions [Ladefoged and Broadbent (1957). J. Acoust. Soc. Am. 29(1), 98-104] or filtered to emphasize narrow [Kiefte and Kluender (2008). J. Acoust. Soc. Am. 123(1), 366-376] or broad frequency regions [Watkins (1991). J. Acoust. Soc. Am. 90(6), 2942-2955]. Spectral differences between filtered precursors and vowel targets are perceptually enhanced, producing spectral contrast effects (e.g., emphasizing spectral properties of /ɪ/ in the precursor elicited more /ɛ/ responses to an /ɪ/-/ɛ/ vowel continuum, and vice versa). Historically, precursors have been processed by high-gain filters, resulting in prominent stable long-term spectral properties. Perceptual sensitivity to subtler but equally reliable spectral properties is unknown. Here, precursor sentences were processed by filters of variable bandwidths and different gains, then followed by vowel sounds varying from /ɪ/-/ɛ/. Contrast effects were widely observed, including when filters had only 100-Hz bandwidth or +5 dB gain. Average filter power was a good predictor of the magnitudes of contrast effects, revealing a close linear correspondence between the prominence of a reliable spectral property and the size of shifts in perceptual responses. High sensitivity to subtle spectral regularities suggests contrast effects are not limited to high-power filters, and thus may be more pervasive in speech perception than previously thought.

Speech perception in simulated electric hearing exploits information-bearing acoustic change

Stilp and Kluender [(2010). Proc. Natl. Acad. Sci. U.S.A. 107(27), 12387-12392] reported measures of sensory change over time (cochlea-scaled spectral entropy, CSE) reliably predicted sentence intelligibility for normal-hearing listeners. Here, implications for listeners with atypical hearing were explored using noise-vocoded speech. CSE was parameterized as Euclidean distances between biologically scaled spectra [measured before sentences were noise vocoded (CSE)] or between channel amplitude profiles in simulated cochlear-implant processing [measured after vocoding (CSE(CI))]. Sentence intelligibility worsened with greater amounts of information replaced by noise; patterns of performance did not differ between CSE and CSE(CI). Results demonstrate the importance of information-bearing change for speech perception in simulated electric hearing.

Information-bearing acoustic change outperforms duration in predicting intelligibility of full-spectrum and noise-vocoded sentencesa)

Recent research has demonstrated a strong relationship between information-bearing acoustic changes in the speech signal and speech intelligibility. The availability of information-bearing acoustic changes reliably predicts intelligibility of full-spectrum [Stilp and Kluender (2010). Proc. Natl. Acad. Sci. U.S.A. 107(27), 12387-12392] and noise-vocoded sentences amid noise interruption [Stilp et al. (2013). J. Acoust. Soc. Am. 133(2), EL136-EL141]. However, other research reports that proportion of signal duration preserved also predicts intelligibility of noise-interrupted speech. These factors have only ever been investigated independently, obscuring whether one better explains speech perception. The present experiments manipulated both factors to answer this question. A broad range of sentence durations (160-480 ms) containing high or low information-bearing acoustic changes were replaced by speech-shaped noise in noise-vocoded (Experiment 1) and full-spectrum sentences (Experiment 2). Sentence intelligibility worsened with increasing noise replacement, but in both experiments, information-bearing acoustic change was a statistically superior predictor of performance. Perception relied more heavily on information-bearing acoustic changes in poorer listening conditions (in spectrally degraded sentences and amid increasing noise replacement). Highly linear relationships between measures of information and performance suggest that exploiting information-bearing acoustic change is a shared principle underlying perception of acoustically rich and degraded speech. Results demonstrate the explanatory power of information-theoretic approaches for speech perception.

Perception of Vowel Sounds Within a Biologically Realistic Model of Efficient Coding

Predicated upon principles of information theory, efficient coding has proven valuable for understanding visual perception. Here, we illustrate how efficient coding provides a powerful explanatory framework for understanding speech perception. This framework dissolves debates about objects of perception, instead focusing on the objective of perception: optimizing information transmission between the environment and perceivers. A simple measure of physiologically significant information is shown to predict intelligibility of variable-rate speech and discriminability of vowel sounds. Reliable covariance between acoustic attributes in complex sounds, both speech and nonspeech, is demonstrated to be amply available in natural sounds and efficiently coded by listeners. An efficient coding framework provides a productive approach to answer questions concerning perception of vowel sounds (including vowel inherent spectral change), perception of speech, and perception most broadly.

Modest, reliable spectral peaks in preceding sounds influence vowel perception

When a spectral property is reliable across an acoustic context and subsequent vowel target, perception deemphasizes this cue and shifts toward less predictable, more informative cues. This phenomenon (auditory perceptual calibration) has been demonstrated for reliable spectral peaks +20 dB or larger, but psychoacoustic findings predict sensitivity to more modest spectral peaks. Listeners identified vowel targets following a sentence with a reliable +2 to +15 dB spectral peak centered at F2 of the vowel. Vowel identifications weighted F2 significantly less when reliable peaks were at least +5 dB. Results demonstrate high sensitivity to reliable acoustic properties in the sensory environment.

Efficient coding and statistically optimal weighting of covariance among acoustic attributes in novel sounds.

To the extent that sensorineural systems are efficient, redundancy should be extracted to optimize transmission of information, but perceptual evidence for this has been limited. Stilp and colleagues recently reported efficient coding of robust correlation (r = .97) among complex acoustic attributes (attack/decay, spectral shape) in novel sounds. Discrimination of sounds orthogonal to the correlation was initially inferior but later comparable to that of sounds obeying the correlation. These effects were attenuated for less-correlated stimuli (r = .54) for reasons that are unclear. Here, statistical properties of correlation among acoustic attributes essential for perceptual organization are investigated. Overall, simple strength of the principal correlation is inadequate to predict listener performance. Initial superiority of discrimination for statistically consistent sound pairs was relatively insensitive to decreased physical acoustic/psychoacoustic range of evidence supporting the correlation, and to more frequent presentations of the same orthogonal test pairs. However, increased range supporting an orthogonal dimension has substantial effects upon perceptual organization. Connectionist simulations and Eigenvalues from closed-form calculations of principal components analysis (PCA) reveal that perceptual organization is near-optimally weighted to shared versus unshared covariance in experienced sound distributions. Implications of reduced perceptual dimensionality for speech perception and plausible neural substrates are discussed.

Statistical structure of speech sound classes is congruent with cochlear nucleus response properties

Natural sounds possess considerable statistical structure. Lewicki (2002, Nature Neuroscience, 5(4):356-363) used independent components analysis (ICA) to reveal the statistical structure of environmental sounds, animal vocalizations, and human speech. Each sound class exhibited distinct statistical properties, but filters that optimally encoded speech closely resembled response properties in the mammalian auditory nerve. This and other analyses of statistical properties of speech examine only global structure without considering systematic variability in different speech sound classes, while acoustic/phonetic analyses of these classes are agnostic to their statistical structure. Here, statistical structure was investigated in principled subdivisions of speech: consonants organized by manner of articulation, and vowels organized by vocal tract configuration. Analyses reveal systematic differences for local statistical structure in speech: statistically optimal filters in ICA were highly diverse for differ...