Kaibao Nie

Encoding frequency Modulation to improve cochlear implant performance in noise

Different from traditional Fourier analysis, a signal can be decomposed into amplitude and frequency modulation components. The speech processing strategy in most modern cochlear implants only extracts and encodes amplitude modulation in a limited number of frequency bands. While amplitude modulation encoding has allowed cochlear implant users to achieve good speech recognition in quiet, their performance in noise is severely compromised. Here, we propose a novel speech processing strategy that encodes both amplitude and frequency modulations in order to improve cochlear implant performance in noise. By removing the center frequency from the subband signals and additionally limiting the frequency modulations range and rate, the present strategy transforms the fast-varying temporal fine structure into a slowly varying frequency modulation signal. As a first step, we evaluated the potential contribution of additional frequency modulation to speech recognition in noise via acoustic simulations of the cochlear implant. We found that while amplitude modulation from a limited number of spectral bands is sufficient to support speech recognition in quiet, frequency modulation is needed to support speech recognition in noise. In particular, improvement by as much as 71 percentage points was observed for sentence recognition in the presence of a competing voice. The present result strongly suggests that frequency modulation be extracted and encoded to improve cochlear implant performance in realistic listening situations. We have proposed several implementation methods to stimulate further investigation.

On the dichotomy in auditory perception between temporal envelope and fine structure cues (L)

It is important to know what cues the sensory system extracts from natural stimuli and how the brain uses them to form perception. To explore this issue, Smith, Delgutte, and Oxenham [Nature (London) 416, 87–90 (2002)] mixed one sound’s temporal envelope with another sound’s fine temporal structure to produce auditory chimaeras and found that “the perceptual importance of the envelope increases with the number of frequency bands, while that of the fine structure diminishes.” This study addressed two technical issues related to natural cochlear filtering and artificial filter ringing in the chimaerizing algorithm. In addition, this study found that the dichotomy in auditory perception revealed by auditory chimaeras is an epiphenomenon of the classic dichotomy between low- and high-frequency processing. Finally, this study found that the temporal envelope determines sound location as long as the interaural level difference cue is present. The present result reinforces the original hypothesis that the tempor...

Spectral and Temporal Cues in Cochlear Implant Speech Perception

Objective: Taking advantage of the flexibility in the number of stimulating electrodes and the stimulation rate in a modern cochlear implant, the present study evaluated relative contributions of spectral and temporal cues to cochlear implant speech perception. Design: Four experiments were conducted by using a Research Interface Box in five MED-EL COMBI 40+ cochlear implant users. Experiment 1 varied the number of electrodes from four to twelve or the maximal number of available active electrodes while keeping a constant stimulation rate at 1000 Hz per electrode. Experiment 2 varied the stimulation rate from 1000 to 4000 Hz per electrode on four pairs of fixed electrodes. Experiment 3 covaried the number of stimulating electrodes and the stimulation rate to study the trade-off between spectral and temporal cues. Experiment 4 studied the effects of envelope extraction on speech perception and listening preference, including half-wave rectification, full-wave rectification, and the Hilbert transform. Vowels, consonants, and HINT sentences in quiet, as well as with a competing female voice served as test materials. Results: Experiment 1 found significant improvement in all speech tests with a higher number of stimulating electrodes. Experiment 2 found a significant advantage of the high stimulation rate only on consonant recognition and sentence recognition in noise. Experiment 3 found an almost linear trade-off between the number of stimulation electrodes and the stimulation rate for consonant and sentence recognition in quiet, but not for vowel and sentence recognition in noise. Experiment 4 found significantly better performance with the Hilbert transform and the full-wave rectification than the half-wave rectification. In addition, envelope extraction with the Hilbert transform produced the highest rating on subjective judgment of sound quality. Conclusions: Consistent with previous studies, the present result from the five MED-EL subjects showed that (1) the temporal envelope cues from a limited number of channels are sufficient to support high levels of phoneme and sentence recognition in quiet but not for speech recognition in a competing voice, (2) consonant recognition relies more on temporal cues while vowel recognition relies more on spectral cues, (3) spectral and temporal cues can be traded to some degree to produce similar performance in cochlear implant speech recognition, and (4) the Hilbert envelope improves both speech intelligibility and quality in cochlear implants.

Development and validation of the University of Washington Clinical Assessment of Music Perception test.

Objectives: Assessment of cochlear implant outcomes centers around speech discrimination. Despite dramatic improvements in speech perception, music perception remains a challenge for most cochlear implant users. No standardized test exists to quantify music perception in a clinically practical manner. This study presents the University of Washington Clinical Assessment of Music Perception (CAMP) test as a reliable and valid music perception test for English-speaking, adult cochlear implant users. Design: Forty-two cochlear implant subjects were recruited from the University of Washington Medical Center cochlear implant program and referred by two implant manufacturers. Ten normal-hearing volunteers were drawn from the University of Washington Medical Center and associated campuses. A computer-driven, self-administered test was developed to examine three specific aspects of music perception: pitch direction discrimination, melody recognition, and timbre recognition. The pitch subtest used an adaptive procedure to determine just-noticeable differences for complex tone pitch direction discrimination within the range of 1 to 12 semitones. The melody and timbre subtests assessed recognition of 12 commonly known melodies played with complex tones in an isochronous manner and eight musical instruments playing an identical five-note sequence, respectively. Testing was repeated for cochlear implant subjects to evaluate test-retest reliability. Normal-hearing volunteers were also tested to demonstrate differences in performance in the two populations. Results: For cochlear implant subjects, pitch direction discrimination just-noticeable differences ranged from 1 to 8.0 semitones (Mean = 3.0, SD = 2.3). Melody and timbre recognition ranged from 0 to 94.4% correct (mean = 25.1, SD = 22.2) and 20.8 to 87.5% (mean = 45.3, SD = 16.2), respectively. Each subtest significantly correlated at least moderately with both Consonant-Nucleus-Consonant (CNC) word recognition scores and spondee recognition thresholds in steady state noise and two-talker babble. Intraclass coefficients demonstrating test-retest correlations for pitch, melody, and timbre were 0.85, 0.92, and 0.69, respectively. Normal-hearing volunteers had a mean pitch direction discrimination threshold of 1.0 semitone, the smallest interval tested, and mean melody and timbre recognition scores of 87.5 and 94.2%, respectively. Conclusions: The CAMP test discriminates a wide range of music perceptual ability in cochlear implant users. Moderate correlations were seen between music test results and both Consonant-Nucleus-Consonant word recognition scores and spondee recognition thresholds in background noise. Test-retest reliability was moderate to strong. The CAMP test provides a reliable and valid metric for a clinically practical, standardized evaluation of music perception in adult cochlear implant users.

A novel speech-processing strategy incorporating tonal information for cochlear implants

Good performance in cochlear implant users depends in large part on the ability of a speech processor to effectively decompose speech signals into multiple channels of narrow-band electrical pulses for stimulation of the auditory nerve. Speech processors that extract only envelopes of the narrow-band signals (e.g., the continuous interleaved sampling (CIS) processor) may not provide sufficient information to encode the tonal cues in languages such as Chinese. To improve the performance in cochlear implant users who speak tonal language, we proposed and developed a novel speech-processing strategy, which extracted both the envelopes of the narrow-band signals and the fundamental frequency (F/sub 0/) of the speech signal, and used them to modulate both the amplitude and the frequency of the electrical pulses delivered to stimulation electrodes. We developed an algorithm to extract the fundamental frequency and identified the general patterns of pitch variations of four typical tones in Chinese speech. The effectiveness of the extraction algorithm was verified with an artificial neural network that recognized the tonal patterns from the extracted F/sub 0/ information. We then compared the novel strategy with the envelope-extraction CIS strategy in human subjects with normal hearing. The novel strategy produced significant improvement in perception of Chinese tones, phrases, and sentences. This novel processor with dynamic modulation of both frequency and amplitude is encouraging for the design of a cochlear implant device for sensorineurally deaf patients who speak tonal languages.

Contribution of frequency modulation to speech recognition in noise.

Cochlear implants allow most patients with profound deafness to successfully communicate under optimal listening conditions. However, the amplitude modulation (AM) information provided by most implants is not sufficient for speech recognition in realistic settings where noise is typically present. This study added slowly varying frequency modulation (FM) to the existing algorithm of an implant simulation and used competing sentences to evaluate FM contributions to speech recognition in noise. Potential FM advantage was evaluated as a function of the number of spectral bands, FM depth, FM rate, and FM band distribution. Barring floor and ceiling effects, significant improvement was observed for all bands from 1 to 32 with the additional FM cue both in quiet and noise. Performance also improved with greater FM depth and rate, which might reflect resolved sidebands under the FM condition. Having FM present in low-frequency bands was more beneficial than in high-frequency bands, and only half of the bands required the presence of FM, regardless of position, to achieve performance similar to when all bands had the FM cue. These results provide insight into the relative contributions of AM and FM to speech communication and the potential advantage of incorporating FM for cochlear implant signal processing.

Implantation of the Semicircular Canals with Preservation of Hearing and Rotational Sensitivity: a vestibular neurostimulator suitable for clinical research

Hypothesis It is possible to implant a stimulating electrode array in the semicircular canals without damaging rotational sensitivity or hearing. The electrodes will evoke robust and precisely controlled eye movements. Background A number of groups are attempting to develop a neural prosthesis to ameliorate abnormal vestibular function. Animal studies demonstrate that electrodes near the canal ampullae can produce electrically evoked eye movements. The target condition of these studies is typically bilateral vestibular hypofunction. Such a device could potentially be more widely useful clinically and would have a simpler roadmap to regulatory approval if it produced minimal or no damage to the native vestibular and auditory systems. Methods An electrode array was designed for insertion into the bony semicircular canal adjacent to the membranous canal. It was designed to be sufficiently narrow so as to not compress the membranous canal. The arrays were manufactured by Cochlear, Ltd., and linked to a Nucleus Freedom receiver/stimulator. Seven behaviorally trained rhesus macaques had arrays placed in 2 semicircular canals using a transmastoid approach and “soft surgical” procedures borrowed from Hybrid cochlear implant surgery. Postoperative vestibulo-ocular reflex was measured in a rotary chair. Click-evoked auditory brainstem responses were also measured in the 7 animals using the contralateral ear as a control. Results All animals had minimal postoperative vestibular signs and were eating within hours of surgery. Of 6 animals tested, all had normal postoperative sinusoidal gain. Of 7 animals, 6 had symmetric postoperative velocity step responses toward and away from the implanted ear. The 1 animal with significantly asymmetric velocity step responses also had a significant sensorineural hearing loss. One control animal that underwent canal plugging had substantial loss of the velocity step response toward the canal-plugged ear. In 5 animals, intraoperative electrically evoked vestibular compound action potential recordings facilitated electrode placement. Postoperatively, electrically evoked eye movements were obtained from electrodes associated with an electrically evoked vestibular compound action potential wave form. Hearing was largely preserved in 6 animals and lost in 1 animal. Conclusion It is possible to implant the vestibular system with prosthetic stimulating electrodes without loss of rotational sensitivity or hearing. Because electrically evoked eye movements can be reliably obtained with the assistance of intraoperative electrophysiology, it is appropriate to consider treatment of a variety of vestibular disorders using prosthetic electrical stimulation. Based on these findings, and others, a feasibility study for the treatment of human subjects with disabling Ménière’s disease has begun.

Relationship Between Behavioral and Physiological Spectral-Ripple Discrimination

Previous studies have found a significant correlation between spectral-ripple discrimination and speech and music perception in cochlear implant (CI) users. This relationship could be of use to clinicians and scientists who are interested in using spectral-ripple stimuli in the assessment and habilitation of CI users. However, previous psychoacoustic tasks used to assess spectral discrimination are not suitable for all populations, and it would be beneficial to develop methods that could be used to test all age ranges, including pediatric implant users. Additionally, it is important to understand how ripple stimuli are processed in the central auditory system and how their neural representation contributes to behavioral performance. For this reason, we developed a single-interval, yes/no paradigm that could potentially be used both behaviorally and electrophysiologically to estimate spectral-ripple threshold. In experiment 1, behavioral thresholds obtained using the single-interval method were compared to thresholds obtained using a previously established three-alternative forced-choice method. A significant correlation was found (r = 0.84, p = 0.0002) in 14 adult CI users. The spectral-ripple threshold obtained using the new method also correlated with speech perception in quiet and noise. In experiment 2, the effect of the number of vocoder-processing channels on the behavioral and physiological threshold in normal-hearing listeners was determined. Behavioral thresholds, using the new single-interval method, as well as cortical P1-N1-P2 responses changed as a function of the number of channels. Better behavioral and physiological performance (i.e., better discrimination ability at higher ripple densities) was observed as more channels added. In experiment 3, the relationship between behavioral and physiological data was examined. Amplitudes of the P1-N1-P2 “change” responses were significantly correlated with d′ values from the single-interval behavioral procedure. Results suggest that the single-interval procedure with spectral-ripple phase inversion in ongoing stimuli is a valid approach for measuring behavioral or physiological spectral resolution.

Feasibility of Single Channel Speaker Separation Based on Modulation Frequency Analysis

We explore the use of the modulation frequency domain for single channel speaker separation. We discuss features of the modulation spectrogram of speech signals that suggest that multiple speakers are highly separable in this space. In a preliminary experiment, we separate a target speaker from an interfering speaker by manually masking out modulation spectral features of the interferer. We extend this experiment into a new automatic speaker separation algorithm, and show that it achieves an acceptable level of separation. The new algorithm only needs a rough estimate of the target speakers pitch range.

Prosthetic implantation of the human vestibular system

Hypothesis A functional vestibular prosthesis can be implanted in human such that electrical stimulation of each semicircular canal produces canal-specific eye movements while preserving vestibular and auditory function. Background A number of vestibular disorders could be treated with prosthetic stimulation of the vestibular end organs. We have previously demonstrated in rhesus monkeys that a vestibular neurostimulator, based on the Nucleus Freedom cochlear implant, can produce canal-specific electrically evoked eye movements while preserving auditory and vestibular function. An investigational device exemption has been obtained from the FDA to study the feasibility of treating uncontrolled Ménière’s disease with the device. Methods The UW/Nucleus vestibular implant was implanted in the perilymphatic space adjacent to the three semicircular canal ampullae of a human subject with uncontrolled Ménière’s disease. Preoperative and postoperative vestibular and auditory function was assessed. Electrically evoked eye movements were measured at 2 time points postoperatively. Results Implantation of all semicircular canals was technically feasible. Horizontal canal and auditory function were largely, but not totally, lost. Electrode stimulation in 2 of 3 canals resulted in canal-appropriate eye movements. Over time, stimulation thresholds increased. Conclusion Prosthetic implantation of the semicircular canals in humans is technically feasible. Electrical stimulation resulted in canal-specific eye movements, although thresholds increased over time. Preservation of native auditory and vestibular function, previously observed in animals, was not demonstrated in a single subject with advanced Ménière’s disease.