Jennifer M. Fellowes

Talker identification based on phonetic information

Accounts of the identification of words and talkers commonly rely on different acoustic properties. To identify a word, a perceiver discards acoustic aspects of an utterance that are talker specific, forming an abstract representation of the linguistic message with which to probe a mental lexicon. To identify a talker, a perceiver discards acoustic aspects of an utterance specific to particular phonemes, creating a representation of voice quality with which to search for familiar talkers in long-term memory. In 3 experiments, sinewave replicas of natural speech sampled from 10 talkers eliminated natural voice quality while preserving idiosyncratic phonetic variation. Listeners identified the sinewave talkers without recourse to acoustic attributes of natural voice quality. This finding supports a revised description of speech perception in which the phonetic properties of utterances serve to identify both words and talkers.

Learning to Recognize Talkers From Natural, Sinewave, and Reversed Speech Samples

In 5 experiments, the authors investigated how listeners learn to recognize unfamiliar talkers and how experience with specific utterances generalizes to novel instances. Listeners were trained over several days to identify 10 talkers from natural, sinewave, or reversed speech sentences. The sinewave signals preserved phonetic and some suprasegmental properties while eliminating natural vocal quality. In contrast, the reversed speech signals preserved vocal quality while distorting temporally based phonetic properties. The training results indicate that listeners learned to identify talkers even from acoustic signals lacking natural vocal quality. Generalization performance varied across the different signals and depended on the salience of phonetic information. The results suggest similarities in the phonetic attributes underlying talker recognition and phonetic perception.

Perceiving the sex and identity of a talker without natural vocal timbre

The personal attributes of a talker perceived via acoustic properties of speech are commonly considered to be an extralinguistic message of an utterance. Accordingly, accounts of the perception of talker attributes have emphasized a causal role of aspects of the fundamental frequency and coarsegrain acoustic spectra distinct from the detailed acoustic correlates of phonemes. In testing this view, in four experiments, we estimated the ability of listeners to ascertain the sex or the identity of 5 male and 5 female talkers from sinusoidal replicas of natural utterances, which lack fundamental frequency and natural vocal spectra. Given such radically reduced signals, listeners appeared to identify a talker’s sex according to the central spectral tendencies of the sinusoidal constituents. Under acoustic conditions that prevented listeners from determining the sex of a talker, individual identification from sinewave signals was often successful. These results reveal that the perception of a talker’s sex and identity are not contingent and that fine-grain aspects of a talker’s phonetic production can elicit individual identification under conditions that block the perception of voice quality.

Multimodal perceptual organization of speech: evidence from tone analogs of spoken utterances

Theoretical and practical motives alike have prompted recent investigations of multimodal speech perception. Theoretically, multimodal studies have extended the conceptualization of perceptual organization beyond the familiar modality-bound accounts deriving from Gestalt psychology. Practically, such investigations have been driven by a need to understand the proficiency of multimodal speech perception using an electrocochlear prosthesis for hearing. In each domain, studies have shown that perceptual organization of speech can occur even when the perceivers auditory experience departs from natural speech qualities. Accordingly, our research examined auditor-visual multimodal integration of videotaped faces and selected acoustic constituents of speech signals, each realized as a single sinewave tone accompanying a video image of an articulating face. The single tone reproduced the frequency and amplitude of the phonatory cycle or of one of the lower three oral formants. Our results showed a distinct advantage for the condition pairing the video image of the face with a sinewave replicating the second formant, despite its unnatural timbre and its presentation in acoustic isolation from the rest of the speech signal. Perceptual coherence of multimodal speech in these circumstances is established when the two modalities concurrently specify the same underlying phonetic attributes.

On the perception of similarity among talkers

A listener who recognizes a talker notices characteristic attributes of the talkers speech despite the novelty of each utterance. Accounts of talker perception have often presumed that consistent aspects of an individuals speech, termed indexical properties, are ascribable to a talkers unique anatomy or consistent vocal posture distinct from acoustic correlates of phonetic contrasts. Accordingly, the perception of a talker is acknowledged to occur independently of the perception of a linguistic message. Alternatively, some studies suggest that attention to attributes of a talker includes indexical linguistic attributes conveyed in the articulation of consonants and vowels. This investigation sought direct evidence of attention to phonetic attributes of speech in perceiving talkers. Natural samples and sinewave replicas derived from them were used in three experiments assessing the perceptual properties of natural and sine-wave sentences; of temporally veridical and reversed natural and sine-wave sentences; and of an acoustic correlate of vocal tract scale to judgments of sine-wave talker similarity. The results revealed that the subjective similarity of individual talkers is preserved in the absence of natural vocal quality; and that local phonetic segmental attributes as well as global characteristics of speech can be exploited when listeners notice characteristics of talkers.

Audio-visual speech perception without speech cues

A series of experiments was conducted in which listeners were presented with audio-visual sentences in a transcription task. The visual components of the stimuli consisted of a male talkers face. The acoustic components consisted of: (1) natural speech; (2) envelope-shaped noise which preserved the duration and amplitude of the original speech waveform; and (3) various types of sine wave speech signals that followed the formant frequencies of a natural utterance. Sine wave speech is a skeletonized version of a natural utterance which contains frequency and amplitude variation of the formants, but lacks any fine-grained acoustic structure of speech. Intelligibility of the present set of sine wave sentences was relatively low in contrast to previous findings (Remez, Rubin, Pisoni, and Carrell, 1981). However, intelligibility was greatly increased when visual information from a talkers face was presented along with the auditory stimuli. Further experiments demonstrated that the intelligibility of single tones increased differentially depending on which formant analog was presented. It was predicted that the increase in intelligibility for the sine wave speech with an added video display would be greater than the gain observed with envelope-shaped noise. This prediction is based on the assumption that the information-bearing phonetic properties of spoken utterances are preserved in the audio-visual sine wave conditions.

Audio-Visual Speech Perception Without Speech Cues: A First Report

A primary objective of a theory of audio-visual speech perception is to describe the process of audio-visual integration and the form of the auditory and visual streams of information. An experiment was conducted in which listeners were presented with audio-visual sentences in a transcription task. The visual components of the stimuli consisted of a face of a male talker. The acoustic components of the audio-visual stimuli consisted of: (1) natural speech (2) envelope-shaped noise which preserved the duration and amplitude of the original speech waveform and (3) various types of sinewave speech signals which preserved different aspects of the time-varying spectrum of the original speech signals. Sinewave speech is a skeletonized version of a natural utterance which contains frequency and amplitude variation of the formants, but lacks any fine-grained acoustic structure of speech. When all three formants are represented in this form (T1+T2+T3) and listeners are told they are listening to speech, the intelligibility of sentences is relatively high (above 75%) (Remez, Rubin, Pisoni, & Carrell 1981). However, when listeners are presented with only single tones (Ti, T2, or T3) performance falls to almost zero. Preliminary results reported here indicate that intelligibility of sine-wave sentences is greatly increased when visual information is combined with the auditory signal. We predicted that the increase in intelligibility for the sinewave speech with an added video display would be greater than the gain observed with the envelope-shaped noise. This prediction is based on the assumption that the phonetic properties of spoken utterances are retained in the audio-visual stream of the sine-wave condition. The results demonstrate that visual information significantly increases the intelligibility of the tonal analog of the second formant, but not the tonal analog of the first formant or the bit-flipped noise. Suggesting that the information contained in the tone 2 analog is useful for audio-visual integration. Thus, the dynamic time-varying properties of the vocal tract transfer function that are encoded in both the optical and acoustic signals play an important role in speech intelligibility, and therefore need to be incorporated in theoretical accounts of audio-visual speech perception.

On the distinctive pitch of vowels: Perceptual prototypes for sinewave analogs?

A recent study by Kuhl et al. (1991) found striking perceptual correspondences between vowels and steady tones. Whether subjects experienced spoken vowels, visually presented images of articulating faces producing vowels, or imaginary vowels, they matched the vowel /ɑ / with a low‐pitch tone, and /i/ with a high‐pitch tone. However, tonal analogs of vowels were matched in the opposite manner, with low pitch associated with the vowel /i/ and high with the vowel /ɑ /. These sine‐wave vowels were therefore excluded from hypothesized recognition mechanisms employing distinctive vowel pitches as perceptual prototypes. This finding is counterevidence to claims that tonal analogs of utterances are perceived through ordinary means. The present study employed sinewave realizations of several words differing solely in the nuclear vowel, /ɑ / or /i/, in an attempt to replicate and extend this finding. Subjects were asked to match the predominant pitch or vowel quality of these medial sine‐wave vowels to the pitch of...

Perceiving the difference between spontaneous and read speech: The role of physical duration.

Despite the greater average duration of spontaneous sentences relative to matched fluently read sentences, it is difficult to identify the physical basis for determining whether an utterance is spontaneous or read. The present experiment compared judgments of spontaneity with judgments of duration for the same speech samples. The test material comprised 25 sentence pairs in which one sentence was spoken spontaneously and one sentence was read, both produced by the same speaker. On each trial, subjects identified the sentence of the pair that was spoken spontaneously, and the sentence that was longer in duration; tests were blocked by judgment type. The greater the absolute duration difference was between the two sentences, the better subjects performed on the duration judgment. The physical differences in duration did not predict the spontaneity judgment, nor did the duration judgment correlate with the spontaneity judgment. The results suggest that physical duration alone is not an effective attribute un...

Phonetic sensitivity and individual recognition: Notes on system architecture

A perceiver’s ability to identify a familiar talker is often ascribed to the registration of acoustic attributes of vocal quality. In idealizations of this aspect of speech perception, unique long‐term characteristics of the glottal source or the supralaryngeal filter of speaking acquaintances are represented in a mental gallery, and function as standards for evaluating unknown signals that challenge the auditory system. The ability to identify a message spoken by a familiar talker inheres in a different set of acoustic properties, those of finer grain that underlie the perception of consonant and vowel sequences useful for lexical retrieval. Neuropsychological reports of a dissociation between aphasia and phonagnosia suggest a system architecture in which the perception of a linguistic message is functionally independent of the perception of the identity of the talker who produced it. The plausibility of this conceptualization will be discussed in the light of perceptual identification and perceptual lea...