Fan-Gang Zeng

Speech Recognition with Primarily Temporal Cues

Nearly perfect speech recognition was observed under conditions of greatly reduced spectral information. Temporal envelopes of speech were extracted from broad frequency bands and were used to modulate noises of the same bandwidths. This manipulation preserved temporal envelope cues in each band but restricted the listener to severely degraded information on the distribution of spectral energy. The identification of consonants, vowels, and words in simple sentences improved markedly as the number of bands increased; high speech recognition performance was obtained with only three bands of modulated noise. Thus, the presentation of a dynamic temporal pattern in only a few broad spectral regions is sufficient for the recognition of speech.

Cochlear Implants: System Design, Integration, and Evaluation

As the most successful neural prosthesis, cochlear implants have provided partial hearing to more than 120000 persons worldwide; half of which being pediatric users who are able to develop nearly normal language. Biomedical engineers have played a central role in the design, integration and evaluation of the cochlear implant system, but the overall success is a result of collaborative work with physiologists, psychologists, physicians, educators, and entrepreneurs. This review presents broad yet in-depth academic and industrial perspectives on the underlying research and ongoing development of cochlear implants. The introduction accounts for major events and advances in cochlear implants, including dynamic interplays among engineers, scientists, physicians, and policy makers. The review takes a system approach to address critical issues in cochlear implant research and development. First, the cochlear implant system design and specifications are laid out. Second, the design goals, principles, and methods of the subsystem components are identified from the external speech processor and radio frequency transmission link to the internal receiver, stimulator and electrode arrays. Third, system integration and functional evaluation are presented with respect to safety, reliability, and challenges facing the present and future cochlear implant designers and users. Finally, issues beyond cochlear implants are discussed to address treatment options for the entire spectrum of hearing impairment as well as to use the cochlear implant as a model to design and evaluate other similar neural prostheses such as vestibular and retinal implants.

Music perception with temporal cues in acoustic and electric hearing.

Objective The first specific aim of the present study is to compare the ability of normal-hearing and cochlear implant listeners to use temporal cues in three music perception tasks: tempo discrimination, rhythmic pattern identification, and melody identification. The second aim is to identify the relative contribution of temporal and spectral cues to melody recognition in acoustic and electric hearing. Design Both normal-hearing and cochlear implant listeners participated in the experiments. Tempo discrimination was measured in a two-interval forced-choice procedure in which subjects were asked to choose the faster tempo at four standard tempo conditions (60, 80, 100, and 120 beats per minute). For rhythmic pattern identification, seven different rhythmic patterns were created and subjects were asked to read and choose the musical notation displayed on the screen that corresponded to the rhythmic pattern presented. Melody identification was evaluated with two sets of 12 familiar melodies. One set contained both rhythm and melody information (rhythm condition), whereas the other set contained only melody information (no-rhythm condition). Melody stimuli were also processed to extract the slowly varying temporal envelope from 1, 2, 4, 8, 16, 32, and 64 frequency bands, to create cochlear implant simulations. Subjects listened to a melody and had to respond by choosing one of the 12 names corresponding to the melodies displayed on a computer screen. Results In tempo discrimination, the cochlear implant listeners performed similarly to the normal-hearing listeners with rate discrimination difference limens obtained at 4–6 beats per minute. In rhythmic pattern identification, the cochlear implant listeners performed 5–25 percentage points poorer than the normal-hearing listeners. The normal-hearing listeners achieved perfect scores in melody identification with and without the rhythmic cues. However, the cochlear implant listeners performed significantly poorer than the normal-hearing listeners in both rhythm and no-rhythm conditions. The simulation results from normal-hearing listeners showed a relatively high level of performance for all numbers of frequency bands in the rhythm condition but required as many as 32 bands in the no-rhythm condition. Conclusions Cochlear-implant listeners performed normally in tempo discrimination, but significantly poorer than normal-hearing listeners in rhythmic pattern identification and melody recognition. While both temporal (rhythmic) and spectral (pitch) cues contribute to melody recognition, cochlear-implant listeners mostly relied on the rhythmic cues for melody recognition. Without the rhythmic cues, high spectral resolution with as many as 32 bands was needed for melody recognition for normal-hearing listeners. This result indicates that the present cochlear implants provide sufficient spectral cues to support speech recognition in quiet, but they are not adequate to support music perception. Increasing the number of functional channels and improved encoding of the fine structure information are necessary to improve music perception for cochlear implant listeners.

Speech and melody recognition in binaurally combined acoustic and electric hearing

Speech recognition in noise and music perception is especially challenging for current cochlear implant users. The present study utilizes the residual acoustic hearing in the nonimplanted ear in five cochlear implant users to elucidate the role of temporal fine structure at low frequencies in auditory perception and to test the hypothesis that combined acoustic and electric hearing produces better performance than either mode alone. The first experiment measured speech recognition in the presence of competing noise. It was found that, although the residual low-frequency (<1000 Hz) acoustic hearing produced essentially no recognition for speech recognition in noise, it significantly enhanced performance when combined with the electric hearing. The second experiment measured melody recognition in the same group of subjects and found that, contrary to the speech recognition result, the low-frequency acoustic hearing produced significantly better performance than the electric hearing. It is hypothesized that listeners with combined acoustic and electric hearing might use the correlation between the salient pitch in low-frequency acoustic hearing and the weak pitch in the envelope to enhance segregation between signal and noise. The present study suggests the importance and urgency of accurately encoding the fine-structure cue in cochlear implants.

Trends in Cochlear Implants

More than 60,000 people worldwide use cochlear implants as a means to restore functional hearing. Although individual performance variability is still high, an average implant user can talk on the phone in a quiet environment. Cochlear-implant research has also matured as a field, as evidenced by the exponential growth in both the patient population and scientific publication. The present report examines current issues related to audiologic, clinical, engineering, anatomic, and physiologic aspects of cochlear implants, focusing on their psychophysical, speech, music, and cognitive performance. This report also forecasts clinical and research trends related to presurgical evaluation, fitting protocols, signal processing, and postsurgical rehabilitation in cochlear implants. Finally, a future landscape in amplification is presented that requires a unique, yet complementary, contribution from hearing aids, middle ear implants, and cochlear implants to achieve a total solution to the entire spectrum of hearing loss treatment and management.

Cochlear implant speech recognition with speech maskers

Speech recognition performance was measured in normal-hearing and cochlear-implant listeners with maskers consisting of either steady-state speech-spectrum-shaped noise or a competing sentence. Target sentences from a male talker were presented in the presence of one of three competing talkers (same male, different male, or female) or speech-spectrum-shaped noise generated from this talker at several target-to-masker ratios. For the normal-hearing listeners, target-masker combinations were processed through a noise-excited vocoder designed to simulate a cochlear implant. With unprocessed stimuli, a normal-hearing control group maintained high levels of intelligibility down to target-to-masker ratios as low as 0 dB and showed a release from masking, producing better performance with single-talker maskers than with steady-state noise. In contrast, no masking release was observed in either implant or normal-hearing subjects listening through an implant simulation. The performance of the simulation and implant groups did not improve when the single-talker masker was a different talker compared to the same talker as the target speech, as was found in the normal-hearing control. These results are interpreted as evidence for a significant role of informational masking and modulation interference in cochlear implant speech recognition with fluctuating maskers. This informational masking may originate from increased target-masker similarity when spectral resolution is reduced.

Temporal pitch in electric hearing

Both place and temporal codes in the peripheral auditory system contain pitch information, however, their actual use by the brain is unclear. Here pitch data are reported from users of the cochlear implant, which provides the ability to change the temporal code independently from the place code. With fixed electrode stimulation, both frequency discrimination and pitch estimate data show that the cochlear implant users can only discern differences in pitch for frequencies up to about 300 Hz. An integration model can predict pitch estimation from frequency discrimination, reinforcing Fechners hypothesis relating sensation magnitude to stimulus discriminability. The present results suggest that 300 Hz is the upper boundary of the temporal code and that the absolute place information should be included in the present pitch models. They further suggest that future cochlear implants need to increase the number of independent electrodes to restore normal pitch range and resolution.

Importance of tonal envelope cues in Chinese speech recognition

Temporal waveform envelope cues provide significant information for English speech recognition, and, when combined with lip reading, could produce near‐perfect consonant identification performance [Van Tasell et al., 1152–1161 (1987)]. Tonal patterns are important for Chinese speech recognition and can be effectively conveyed by temporal envelope cues [D. H. Whalen and Y. Xu, Phonetics 49, 25–47 (1992)]. This study investigates whether tones can help Chinese‐speaking listeners use envelope cues more effectively than English listeners. The speech envelope was extracted from broad frequency bands and used to modulate a noise of the same bandwidth. Mandarin vowels, consonants, tones, and sentences were identified by ten native Chinese‐speaking listeners with 1, 2, 3, and 4 noise bands (or channels). The results showed that recognition of vowels, consonants and sentences increases dramatically with the number of channels, a pattern similar to that observed in English speech recognition. However, tones were co...

Human hearing enhanced by noise.

Noise was traditionally regarded as a nuisance, which should be minimized if possible. However, recent research has shown that addition of an appropriate amount of noise can actually improve signal detection in a nonlinear system, an effect called stochastic resonance. While stochastic resonance has been described in a variety of physical and biological systems, its functional significance in human sensory systems remains mostly unexplored. Here we report psychophysical data showing that signal detection and discrimination can be enhanced by noise in human subjects whose hearing is evoked by either normal acoustic stimulation or electric stimulation of the auditory nerve or the brainstem. Our results suggest that noise is an integral part of the normal sensory process and should be added to auditory prostheses.

Speech recognition with altered spectral distribution of envelope cues

Recognition of consonants, vowels, and sentences was measured in conditions of reduced spectral resolution and distorted spectral distribution of temporal envelope cues. Speech materials were processed through four bandpass filters (analysis bands), half-wave rectified, and low-pass filtered to extract the temporal envelope from each band. The envelope from each speech band modulated a band-limited noise (carrier bands). Analysis and carrier bands were manipulated independently to alter the spectral distribution of envelope cues. Experiment I demonstrated that the location of the cutoff frequencies defining the bands was not a critical parameter for speech recognition, as long as the analysis and carrier bands were matched in frequency extent. Experiment II demonstrated a dramatic decrease in performance when the analysis and carrier bands did not match in frequency extent, which resulted in a warping of the spectral distribution of envelope cues. Experiment III demonstrated a large decrease in performance when the carrier bands were shifted in frequency, mimicking the basal position of electrodes in a cochlear implant. And experiment IV showed a relatively minor effect of the overlap in the noise carrier bands, simulating the overlap in neural populations responding to adjacent electrodes in a cochlear implant. Overall, these results show that, for four bands, the frequency alignment of the analysis bands and carrier bands is critical for good performance, while the exact frequency divisions and overlap in carrier bands are not as critical.