Joshua D. Reiss

Intelligent systems for mixing multichannel audio

Multichannel signal processing techniques are usually concerned with extracting information about sources from several received signals. In this paper, we describe an emerging field of multichannel audio signal processing where the inter-channel relationships are exploited in order to manipulate the multichannel content. Applications to real-time, automatic audio production are described and the necessary technologies and the architecture of such systems are presented. The current state of the art is reviewed, and directions of future research are also discussed.

Self-Localization of Ad-Hoc Arrays Using Time Difference of Arrivals

We investigate the problem of sensor and source joint localization using time-difference of arrivals (TDOAs) of an ad-hoc array. A major challenge is that the TDOAs contain unknown time offsets between asynchronous sensors. To address this problem, we propose a low-rank approximation method that does not need any prior knowledge of sensor and source locations or timing information. At first, we construct a pseudo time of arrival (TOA) matrix by introducing two sets of unknown timing parameters (source onset times and device capture times) into the current TDOA matrix. Then we propose a Gauss-Newton low-rank approximation algorithm to jointly identify the two sets of unknown timing parameters, exploiting the low-rank property embedded in the pseudo TOA matrix. We derive the boundaries of the timing parameters to reduce the initialization space and employ a multi-initialization scheme. Finally, we use the estimated timing parameters to correct the pseudo TOA matrix, which is further applied to sensor and source localization. Experimental results show that the proposed approach outperforms state-of-the-art algorithms.

Automatic gain and fader control for live mixing

A cross-adaptive mixing device has been developed for the purpose of optimizing the gain levels of a live audio mixture. The method aims to achieve optimal mixing levels by optimizing the ratios between the loudness of each individual input channel and the overall loudness contained in a stereo mix. In order to evaluate the amount of loudness of each channel in real-time, accumulative statistical measurements were performed. The system uses a cross-adaptive algorithm to map the loudness indicators to the channel gain values. The system has applications in automatic mixing of live music, live mixing of game audio, and studio recording post-production.

A real-time semiautonomous audio panning system for music mixing

A real-time semiautonomous stereo panning system for music mixing has been implemented. The system uses spectral decomposition, constraint rules, and cross-adaptive algorithms to perform real-time placement of sources in a stereo mix. A subjective evaluation test was devised to evaluate its quality against human panning. It was shown that the automatic panning technique performed better than a nonexpert and showed no significant statistical difference to the performance of a professional mixing engineer.

Fuzzy Impulsive Control of High-Order Interpolative Low-Pass Sigma–Delta Modulators

In this paper, a fuzzy impulsive control strategy is proposed. The state vectors that the impulsive controller resets to are determined so that the state vectors of interpolative low-pass sigma-delta modulators (SDMs) are bounded within any arbitrary nonempty region no matter what the input step size, the initial condition and the filter parameters are, the occurrence of limit cycle behaviors and the effect of audio clicks are minimized, as well as the state vectors are close to the invariant set if it exists. To work on this problem, first, the local stability criterion and the condition for the occurrence of limit cycle behaviors are derived. Second, based on the derived conditions, as well as a practical consideration based on the boundedness of the state variables and a heuristic measure on the strength of audio clicks, fuzzy membership functions and a fuzzy impulsive control law are formulated. The controlled state vectors are then determined by solving the fuzzy impulsive control law. One of the advantages of the fuzzy impulsive control strategy over the existing linear control strategies is the robustness to the input signal, the initial condition and the filter parameters, and that over the existing nonlinear control strategy are the efficiency and the effectiveness in terms of lower frequency of applying the control force and higher signal-to-noise ratio (SNR) performance

Design of Interpolative Sigma Delta Modulators Via Semi-Infinite Programming

This correspondence considers the optimized design of interpolative sigma delta modulators (SDMs). The first optimization problem is to determine the denominator coefficients. The objective of the optimization problem is to minimize the passband energy of the denominator of the loop filter transfer function (excluding the dc poles) subject to the continuous constraint of this function defined in the frequency domain. The second optimization problem is to determine the numerator coefficients in which the cost function is to minimize the stopband ripple energy of the loop filter subject to the stability condition of the noise transfer function (NTF) and signal transfer function (STF). These two optimization problems are actually quadratic semi-infinite programming (SIP) problems. By employing the dual-parameterization method, global optimal solutions that satisfy the corresponding continuous constraints are guaranteed if the filter length is long enough. The advantages of this formulation are the guarantee of the stability of the transfer functions, applicability to design of rational infinite-impulse-response (IIR) filters without imposing specific filter structures, and the avoidance of iterative design of numerator and denominator coefficients. Our simulation results show that this design yields a significant improvement in the signal-to-noise ratio (SNR) and have a larger stability range, compared with the existing designs

An iterative approach to source counting and localization using two distant microphones

We propose a time difference of arrival (TDOA) estimation framework based on time-frequency inter-channel phase difference (IPD) to count and localize multiple acoustic sources in a reverberant environment using two distant microphones. The time-frequency (T-F) processing enables exploitation of the nonstationarity and sparsity of audio signals, increasing robustness to multiple sources and ambient noise. For inter-channel phase difference estimation, we use a cost function, which is equivalent to the generalized cross correlation with phase transform (GCC) algorithm and which is robust to spatial aliasing caused by large inter-microphone distances. To estimate the number of sources, we further propose an iterative contribution removal (ICR) algorithm to count and locate the sources using the peaks of the GCC function. In each iteration, we first use IPD to calculate the GCC function, whose highest peak is detected as the location of a sound source; then we detect the T-F bins that are associated with this source and remove them from the IPD set. The proposed ICR algorithm successfully solves the GCC peak ambiguities between multiple sources and multiple reverberant paths.

Automatic noise gate settings for drum recordings containing bleed from secondary sources

An algorithm is presented which automatically sets the attack, release, threshold, and hold parameters of a noise gate applied to drum recordings which contain bleed from secondary sources. The gain parameter which controls the amount of attenuation applied when the gate is closed is retained, to allow the user to control the strength of the gate. The gate settings are found by minimising the artifacts introduced to the desirable component of the signal, whilst ensuring that the level of bleed is reduced by a certain amount. The algorithm is tested on kick drum recordings which contain bleed from hi-hats, snare drum, cymbals, and tom toms.

Audio fingerprinting for multi-device self-localization

We investigate the self-localization problem of an ad-hoc network of randomly distributed and independent devices in an open-space environment with low reverberation but heavy noise (e.g. smartphones recording videos of an outdoor event). Assuming a sufficient number of sound sources, we estimate the distance between a pair of devices from the extreme (minimum and maximum) time difference of arrivals (TDOAs) from the sources to the pair of devices without knowing the time offset. The obtained inter-device distances are then exploited to derive the geometrical configuration of the network. In particular, we propose a robust audio fingerprinting algorithm for noisy recordings and perform landmark matching to construct a histogram of the TDOAs of multiple sources. The extreme TDOAs can be estimated from this histogram. By using audio fingerprinting features, the proposed algorithm works robustly in very noisy environments. Experiments with free-field simulation and open-space recordings prove the effectiveness of the proposed algorithm.

Design of Audio Parametric Equalizer Filters Directly in the Digital Domain

Most design procedures for a digital parametric equalizer begin with analog design techniques, followed by applying the bilinear transform to an analog prototype. As an alternative, an approximation to the parametric equalizer is sometimes designed using pole-zero placement techniques. In this paper, we present an exact derivation of the parametric equalizer without resorting to an analog prototype. We show that there are many solutions to the parametric equalizer design constraints as usually stated, but only one of which consistently yields stable, minimum phase behavior with the upper and lower cutoff frequencies positioned around the center frequency. The conditions for complex conjugate poles and zeros are found and the resultant pole zero placements are examined.