Kelly Fitz

Algorithms for computing the time-corrected instantaneous frequency (reassigned) spectrogram, with applications.

A modification of the spectrogram (log magnitude of the short-time Fourier transform) to more accurately show the instantaneous frequencies of signal components was first proposed in 1976 [Kodera et al., Phys. Earth Planet. Inter. 12, 142-150 (1976)], and has been considered or reinvented a few times since but never widely adopted. This paper presents a unified theoretical picture of this time-frequency analysis method, the time-corrected instantaneous frequency spectrogram, together with detailed implementable algorithms comparing three published techniques for its computation. The new representation is evaluated against the conventional spectrogram for its superior ability to track signal components. The lack of a uniform framework for either mathematics or implementation details which has characterized the disparate literature on the schemes has been remedied here. Fruitful application of the method is shown in the realms of speech phonation analysis, whale song pitch tracking, and additive sound modeling.

Separation of components from impulses in reassigned spectrograms

Two computational methods for pruning a reassigned spectrogram to show only quasisinusoidal components, or only impulses, or both, are presented mathematically and provided with step-by-step algorithms. Both methods compute the second-order mixed partial derivative of the short-time Fourier transform phase, and rely on the conditions that components and impulses are each well-represented by reassigned spectrographic points possessing particular values of this derivative. This use of the mixed second-order derivative was introduced by Nelson [J. Acoust. Soc. Am. 110, 2575-2592 (2001)] but here our goals are to completely describe the computation of this derivative in a way that highlights the relations to the two most influential methods of computing a reassigned spectrogram, and also to demonstrate the utility of this technique for plotting spectrograms showing line components or impulses while excluding most other points. When applied to speech signals, vocal tract resonances (formants) or glottal pulsations can be effectively isolated in expanded views of the phonation process.

Analysis/synthesis comparison

We compared six sound analysis/synthesis systems used for computer music. Each system analysed the same collection of twenty-seven varied input sounds, and output the results in Sound Description Interchange Format (SDIF). We describe each system individually then compare the systems in terms of availability, the sound model(s) they use, interpolation models, noise modelling, the mutability of various sound models, the parameters that must be set to perform analysis, and characteristic artefacts. Although we have not directly compared the analysis results among the different systems, our work has made such a comparison possible.

A Spectrogram for the Twenty-First Century

Just as World War II was breaking out in Europe in 1939, a prototype of a remarkable electrical device was being completed at Bell Telephone Laboratories, under the direction of Ralph Potter. This device was able to provide, on a strip of paper, a continuous running document of the Fourier spectrum of a sound signal as it changed through time. Because of the war it was kept under wraps, but its detailed construction and numerous applications were revealed to the scientific community in a series of papers published in the Journal of the Acoustical Society of America (JASA) in 1946, wherein it was called the Sound Spectrograph. The running spectral analysis that it output was termed a spectrogram. The spectrograph has been recorded in history as one of the most useful and influential instruments for acoustic signal processing. In particular, the fields of phonetics and speech communication, which motivated the development of the machine, have been completely transformed by its widespread adoption. Over the decades, the cumbersome and delicate analog spectrograph hardware was transformed into more robust digital hardware at first, and then as computers became generally more powerful, into the digital software incarnations most of us use today. The underlying principle of the spectrogram has never changed; most applied acousticians who do time-frequency analysis are content to use software that in essence simulates the output that appeared 60 years ago in JASA (Fig. 1). Of what else in acoustics can the same be said? Do we use 60-year old microphones? Tape recorders? Loudspeakers? Well, in truth, some of us have not been so content, but a more useful analytical process has never been generally recognized. The rich area of signal proA SPECTROGRAM FOR THE TWENTY-FIRST CENTURY

Evaluation of sound classification algorithms for hearing aid applications

Automatic program switching has been shown to be greatly beneficial for hearing aid users. This feature is mediated by a sound classification system, which is traditionally implemented using simple features and heuristic classification schemes, resulting in an unsatisfactory performance in complex auditory scenarios. In this study, a number of experiments are conducted to systematically assess the impact of more sophisticated classifiers and features on automatic acoustic environment classification performance. The results show that advanced classifiers, such as Hidden Markov Model (HMM) or Gaussian Mixture Model (GMM), greatly improve classification performance over simple classifiers. This change does not require a great increase of computational complexity, provided that a suitable number (5 to 7) of low-level features are carefully chosen. These findings indicate that advanced classifiers can be feasible in hearing aid applications.

Lemur: A bandwidth‐enhanced sinusoidal modeling system

Lemur models sounds as a collection of partials having amplitude, bandwidth, and frequency envelopes. Bandwidth envelopes allow the neat representation of noisy parts of sounds which produce clutter in purely sinusoidal analyses. Lemur associates noise energy with sinusoidal components that are nearby in frequency. Bandwidth envelopes record the fraction of the partial energy that is attributable to noise energy, as opposed to sinusoidal energy. Amplitude envelopes are calculated to include the energy added by the bandwidth association process. When partials are pruned from the model, in the analysis process or during editing, Lemur preserves the overall energy by distributing the removed partials energy among neighboring partials. For sounds with a good spread (in frequency) of partials, Lemur can represent noise energy in the appropriate spectral regions while retaining the manipulability and homogeneity of a purely sinusoidal model. This homogeneity is important for implementing certain kinds of model‐...

Signal Processing in Speech and Hearing Technology

Speech science and technology would scarcely exist today without acoustic signal processing. The same can be said of hearing assistance technology, including hearing aids and cochlear implants. This article will highlight key contributions made by signal processing techniques in the disparate realms of speech analysis, speech recognition, and hearing aids. We can certainly not exhaustively discuss the applications of signal processing in these areas, much less other related fields that are left out entirely, but we hope to provide at the very least a sampling of the wide range of processing techniques that are brought to bear on the various problems in these subfields. While speech itself is an analog signal (or time sequence) of air pressure variations resulting from puffs of air leaving one’s lungs, modulated by the vibrations of one’s vocal cords and filtered by one’s vocal tract, such a vocal signal is normally digitized in most modern applications, including normal telephone lines. The analog-to-digital (A/D) conversion is needed for computer processing, as the analog speech signal (continuous in both time and amplitude), while suitable for one’s ears, is most efficiently handled as a sequence of digital bits. A/D conversion has two parameters: samples/second and bits/sample. The former is specified by the Nyquist rate, twice the highest audio frequency to be preserved in the speech signal, assuming some analog filter suppresses the weaker energy at relatively high frequencies in speech (e.g., above 3.3 kHz in telephone applications, using 8000 samples/s). Like most audible sounds, speech is dominated by energy in the lowest few kHz, but pertinent energy exists to at least 20 kHz, which is why high-quality recordings, such as CDs, sample at rates up to 44.1 kHz. However, speech can be reasonably intelligible even when low-pass filtered to 4 kHz, as the telephone amply demonstrates. Typical speech applications use 16-bit A/D accuracy, although basic logarithmic coding in the telephone shows that 8-bit accuracy can be adequate in many applications, which include automatic speech recognition, where the objective is a mapping into text, rather than a high-quality audio signal to listen to or analyze in depth. Speech spectrum analysis In phonetics and speech science a commonly pursued aim is to analyze the spectrum of speech as completely as possible, to obtain information about the speech articulation and the specific auditory attributes which characterize speech sounds or “phonemes” (consonants and vowels). Spectrum analysis can further our understanding of the variety of sounds in language (a linguistic pursuit), and can also further our understanding of the fundamental nature of normal and disordered speech.

Music through hearing aids: Perception and modeling.

Historically, the primary focus of hearing aid development has been on improving speech perception for those with hearing loss. Modern‐day hearing‐aid wearers, however, face many different types of acoustic signals, such as music, that require different types of processing. Music signals differ from speech signals in a variety of fundamental ways, and relevant perceptual information is conveyed via different signal attributes in the two types of signals. The research described here is an effort to improve music perception in listeners with hearing impairment. First, methods have been developed to quantitatively measure deficits in music perception for impaired and aided listeners. Second, specific perceptual features have been evaluated as to their relative importance in the successful perception of music and that information has been used to guide signal processing development. Finally, the relevant perceptual features have been modeled, and the models have been used to evaluate and compare signal proces...

Multidimensional perceptual scaling of musical timbre by hearing‐impaired listeners.

We examine the impact of hearing loss and hearing aid processing on the perception of musical timbre. Our objective is to identify significant timbre cues for hearing‐impaired listeners, and to assess the impact of hearing aid signal processing on timbre perception. Hearing aids perform dynamic, level‐dependent spectrum shaping that may influence listeners’ perception of musical instrument timbres and their ability to discriminate among them. Grey [“Multidimensional perceptual scaling of musical timbres,” J. Acoust. Soc. Am. 61, 1270 (1977)] showed that sustaining instrument tones equalized for level, loudness, and duration are distinguished primarily along three perceptual dimensions that are strongly correlated with the acoustical dimensions of: (1) spectral energy distribution, (2) spectral fluctuation, and (3) precedent high‐frequency, low‐amplitude energy. Following the work of Grey, we ask listeners having mild to moderately severe sensorineural hearing loss to rate pairs of synthetic musical instru...

Using the reassigned spectrogram to obtain a voiceprint

While established methods for imaging the time‐frequency content of speech—such as the spectrogram—have frequently been christened ‘‘voiceprinting,’’ it is well‐known that it and other currently popular imaging techniques cannot identify an individual’s voice to more than a suggestive extent. The reassigned spectrogram (also known by other names) is a relatively little‐known method [S. A. Fulop and K. Fitz, ‘‘Algorithms for computing the time‐corrected instantaneous frequency (reassigned) spectrogram, with applications,’’ J. Acoust. Soc. Am. 119, 360–377 (2006)] for imaging the time‐frequency spectral information contained in a signal, which is able to show the instantaneous frequencies of signal components as well as the occurrence of impulses with dramatically increased precision compared to the spectrogram (magnitude of the short‐time Fourier transform) or any other energy density time‐frequency representation. It is shown here that it is possible to obtain a reassigned spectrogram image from a person’...