Robert E. Remez

On the perceptual organization of speech

A general account of auditory perceptual organization has developed in the past 2 decades. It relies on primitive devices akin to the Gestalt principles of organization to assign sensory elements to probable groupings and invokes secondary schematic processes to confirm or to repair the possible organization. Although this conceptualization is intended to apply universally, the variety and arrangement of acoustic constituents of speech violate Gestalt principles at numerous junctures, cohering perceptually, nonetheless. The authors report 3 experiments on organization in phonetic perception, using sine wave synthesis to evade the Gestalt rules and the schematic processes alike. These findings falsify a general auditory account, showing that phonetic perceptual organization is achieved by specific sensitivity to the acoustic modulations characteristic of speech signals.

The handbook of speech perception

List of Contributors. Preface: Michael Studdert-Kennedy (Haskins Laboratories). Introduction: David B. Pisoni (Indiana University) and Robert E. Remez (Barnard College). Part I: Sensing Speech. 1. Acoustic Analysis and Synthesis of Speech: James R. Sawusch (University at Buffalo). 2. Perceptual Organization of Speech: Robert E. Remez (Barnard College). 3. Primacy of Multimodal Speech Perception: Lawrence D. Rosenblum (University of California, Riverside). 4. Phonetic Processing by the Speech Perceiving Brain: Lynne E. Bernstein (House Ear Institute). 5. Event-related Evoked Potentials (ERPs) in Speech Perception: Dennis Molfese, Alexandra P. Fonaryova Key, Mandy J. Maguire, Guy O. Dove and Victoria J. Molfese (all University of Louisville). Part II: Perception of Linguistic Properties. 6. Features in Speech Perception and Lexical Access: Kenneth N. Stevens (Massachusetts Institute of Technology). 7. Speech Perception and Phonological Contrast: Edward Flemming (Stanford University). 8. Acoustic Cues to the Perception of Segmental Phonemes: Lawrence J. Raphael (Adelphi University). 9. Clear Speech: Rosalie M. Uchanski (CID at Washington University School of Medicine). 10. Perception of Intonation: Jacqueline Vaissiere (Laboratoire de Phonetique et de Phonologique, Paris). 11. Lexical Stress: Anne C. Cutler (Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands). 12. Slips of the Ear: Z. S. Bond (Ohio University). Part III: Perception of Indexical Properties. 13. Perception of Dialect Variation: Cynthia Clopper and David B. Pisoni (both Indiana University). 14. Perception of Voice Quality: Jody Kreiman (UCLA), Diana Vanlancker-Sidtis (New York University) and Bruce R. Gerratt (UCLA). 15. Speaker Normalization in Speech Perception: Keith A. Johnson (Ohio State University). 16. Perceptual Integration of Linguistic and Non-Linguistic Properties of Speech: Lynne C. Nygaard (Emory University). Part IV: Speech Perception by Special Listeners. 17. Speech Perception in Infants: Derek M. Houston (Indiana University School of Medicine). 18. Speech Perception in Childhood: Amanda C. Walley (University of Alabama, Birmingham). 19. Age-related Changes in Spoken Word Recognition: Mitchell S. Sommers (Washington University). 20. Speech Perception in Deaf Children with Cochlear Implants: David B. Pisoni (Indiana University). 21. Speech Perception following Focal Brain Injury: William Badacker (Johns Hopkins University). 22. Cross-Language Speech Perception: Nuria Sebastian-Galles (Parc Cientific de Barcelona - Hospital de San Joan de Deu). 23. Speech Perception in Specific Language Impairment: Susan Ellis Weismer (University of Wisconsin, Madison). Part V: Recognition of Spoken Words. 24. Spoken Word Recognition: The Challenge of Variation: Paul A. Luce and Conor T. McLennan (State University of New York, Buffalo). 25. Probabilistic Phonotactics in Spoken Word Recognition: Edward T. Auer, Jr. (House Ear Institute) and Paul A. Luce (State University of New York, Buffalo). Part VI: Theoretical Perspectives. 26. The Relation of Speech Perception and Speech Production: Carol A. Fowler and Bruno Galantucci (both Haskins Laboratories). 27. A Neuroethological Perspective on the Perception of Vocal Communication Signals: Timothy Gentner (University of Chicago) and Gregory F. Ball (Johns Hopkins University). Index

Talker identification based on phonetic information

Accounts of the identification of words and talkers commonly rely on different acoustic properties. To identify a word, a perceiver discards acoustic aspects of an utterance that are talker specific, forming an abstract representation of the linguistic message with which to probe a mental lexicon. To identify a talker, a perceiver discards acoustic aspects of an utterance specific to particular phonemes, creating a representation of voice quality with which to search for familiar talkers in long-term memory. In 3 experiments, sinewave replicas of natural speech sampled from 10 talkers eliminated natural voice quality while preserving idiosyncratic phonetic variation. Listeners identified the sinewave talkers without recourse to acoustic attributes of natural voice quality. This finding supports a revised description of speech perception in which the phonetic properties of utterances serve to identify both words and talkers.

Learning to Recognize Talkers From Natural, Sinewave, and Reversed Speech Samples

In 5 experiments, the authors investigated how listeners learn to recognize unfamiliar talkers and how experience with specific utterances generalizes to novel instances. Listeners were trained over several days to identify 10 talkers from natural, sinewave, or reversed speech sentences. The sinewave signals preserved phonetic and some suprasegmental properties while eliminating natural vocal quality. In contrast, the reversed speech signals preserved vocal quality while distorting temporally based phonetic properties. The training results indicate that listeners learned to identify talkers even from acoustic signals lacking natural vocal quality. Generalization performance varied across the different signals and depended on the salience of phonetic information. The results suggest similarities in the phonetic attributes underlying talker recognition and phonetic perception.

On the Bistability of Sine Wave Analogues of Speech

Our studies revealed two stable modes of perceptual organization, one based on attributes of auditory sensory elements and another based on attributes of patterned sensory variation composed by the aggregation of sensory elements. In a dual-task method, listeners attended concurrently to both aspects, component and pattern, of a sine wave analogue of a word. Organization of elements was indexed by several single-mode tests of auditory form perception to verify the perceptual segregation of either an individual formant of a synthetic word or a tonal component of a sinusoidal word analogue. Organization of patterned variation was indexed by a test of lexical identification. The results show the independence of the perception of auditory and phonetic form, which appear to be differently organized concurrent effects of the same acoustic cause.

Perceiving the sex and identity of a talker without natural vocal timbre

The personal attributes of a talker perceived via acoustic properties of speech are commonly considered to be an extralinguistic message of an utterance. Accordingly, accounts of the perception of talker attributes have emphasized a causal role of aspects of the fundamental frequency and coarsegrain acoustic spectra distinct from the detailed acoustic correlates of phonemes. In testing this view, in four experiments, we estimated the ability of listeners to ascertain the sex or the identity of 5 male and 5 female talkers from sinusoidal replicas of natural utterances, which lack fundamental frequency and natural vocal spectra. Given such radically reduced signals, listeners appeared to identify a talker’s sex according to the central spectral tendencies of the sinusoidal constituents. Under acoustic conditions that prevented listeners from determining the sex of a talker, individual identification from sinewave signals was often successful. These results reveal that the perception of a talker’s sex and identity are not contingent and that fine-grain aspects of a talker’s phonetic production can elicit individual identification under conditions that block the perception of voice quality.

Short-Term Reorganization of Auditory Analysis Induced by Phonetic Experience

Sine wave replicas of spoken words can be perceived both as nonphonetic auditory forms and as words, depending on a listeners experience. In this study, brain areas activated by sine wave words were studied with fMRI in two conditions: when subjects perceived the sounds spontaneously as nonphonetic auditory forms (nave condition) and after instruction and brief practice attending to their phonetic attributes (informed condition). The test items were composed such that half replicated natural words (phonetic items) and the other half did not, because the tone analogs of the first and third formants had been temporally reversed (nonphonetic items). Subjects were asked to decide whether an isolated tone analog of the second formant (T2) presented before the sine wave word (T1234) was included in it. Experience in attending to the phonetic properties of the sinusoids interfered with this auditory matching task and was accompanied by a decrease in auditory cortex activation with word replicas but not with the acoustically matched nonphonetic items. Because the activation patterns elicited by equivalent acoustic test items depended on a listeners awareness of their phonetic potential, this indicates that the analysis of speech sounds in the auditory cortex is distinct from the simple resolution of auditory form, and is not a mere consequence of acoustic complexity. Because arbitrary acoustic patterns did not evoke the response observed for phonetic patterns, these findings suggest that the perception of speech is contingent on the presence of familiar patterns of spectral variation. The results are consistent with a short-term functional reorganization of auditory analysis induced by phonetic experience with sine wave replicas and contingent on the dynamic acoustic structure of speech.

Perceptual normalization of vowels produced by sinusoidal voices.

When listeners hear a sinusoidal replica of a sentence, they perceive linguistic properties despite the absence of short-time acoustic components typical of vocal signals. Is this accomplished by a postperceptual strategy that accommodates the anomalous acoustic pattern ad hoc, or is a sinusoidal sentence understood by the ordinary means of speech perception? If listeners treat sinusoidal signals as speech signals however unlike speech they may be, then perception should exhibit the commonplace sensitivity to the dimensions of the originating vocal tract. The present study, employing sinusoidal signals, raised this issue by testing the identification of target /bVt/, or b-vowel-t, syllables occurring in sentences that differed in the range of frequency variation of their component tones. Vowel quality of target syllables was influenced by this acoustic correlate of vocal-tract scale, implying that the perception of these nonvocal signals includes a process of vocal-tract scale, implying that the perception of these nonvocal signals includes a process of vocal-tract normalization. Converging evidence suggests that the perception of sinusoidal vowels depends on the relation among component tones and not on the phonetic likeness of each tone in isolation. The findings support the general claim that sinusoidal replicas of natural speech signals are perceptible phonetically because they preserve time-varying information present in natural signals.

On the perception of intonation from sinusoidal sentences

Listeners can perceive the phonetic value of sinusoidal imitations of speech. These tonal replicas are made by setting time-varying sinusoids equal in frequency and amplitude to the computed peaks of the first three formants of natural utterances. Like formant frequencies, the three sinusoids composing the tonal signal are not necessarily related harmonically, and therefore are unlikely to possess a common fundamental frequency. Moreover, none of the tones falls within the frequency range typical of the fundamental frequency of phonation of the natural utterances upon which sinusoidal signals are based. Naive subjects nevertheless report that intelligible tonal replicas of sentences exhibit unusual “vocal” pitch variation, or intonation. The present study attempted to determine the acoustic basis for this apparent intonation of sinusoidal signals by employing several tests of perceived similarity. Listeners judged the tone corresponding to the first formant to be more like the intonation pattern of a sinusoidal sentence than: (1) a tone corresponding to the second or third formant; (2) a tone presenting the computed missing fundamental of the three tones; or (3) a tone following a plausible fundamental frequency contour generated from the amplitude envelope of the signal. Additionslly, the tone reproducing the first formant pattern was responsible for apparent intonation, even when it occurred in conjunction with a lower tone representing the fundamental frequency pattern of the natural utterance on which the replica was modeled. The effects were not contingent on relative tone amplitude within the sentence replica. The case of sinusoidal sentence “pitch” resembles the phenomenon ofdominance, that is, the general salience of waveform periodicity in the region of 400-1000 Hz for perception of the pitch of complex signals.

The Perception of Speech

Publisher Summary None of the acoustic constituents of speech is unique to speech, although some features of speech are characteristic: a cyclical rise and fall of energy associated with a train of syllables, amplitude peaks, and valleys in the short-term spectrum, and variation over time in the frequency at which the peaks and valleys occur. Despite all, a perceiver often tracks the speech of a specific talker sampling by ear and eye, two kinds of perceptual organization that also combine multimodally and resolves the linguistic properties in the sensory effects. That is to say, perceptual analysis of the symbolic properties of speech succeeds. To gauge the means of resolving the sound produced by a single individual, the contrast between visual and auditory attention is instructive. In attending to a visible object or event, a perceiver typically turns to face it bringing the light reflected by the object of interest to the fovea of the retina. A listeners attention to the audible world achieves spatial and spectral focus psychologically, without the selective benefit of a heading at which auditory pattern acuity peaks. Despite all, perception often reciprocates the patterned variation of a speech stream with its discontinuities—that is, dissimilarities among components, and similarities among its components and those of unattended utterances and other events. This perceptual function is fast, unlearned, keyed to complex patterns of sensory variation, tolerant of anomalous sensory quality, nonsymbolic, and dependent on attention whether elicited or exerted.