Matthew D. Hoffman

Static and Dynamic Source Separation Using Nonnegative Factorizations: A unified view

Source separation models that make use of nonnegativity in their parameters have been gaining increasing popularity in the last few years, spawning a significant number of publications on the topic. Although these techniques are conceptually similar to other matrix decompositions, they are surprisingly more effective in extracting perceptually meaningful sources from complex mixtures. In this article, we will examine the various methodologies and extensions that make up this family of approaches and present them under a unified framework. We will begin with a short description of the basic concepts and in the subsequent sections we will delve in more details and explore some of the latest extensions.

VFerret: content-based similarity search tool for continuous archived video

This paper describes VFerret, a content-based similarity search tool for continuous archived video. Instead of depending on attributes or annotations to search desired data from long-time archived video, our system allows users to perform content-based similarity search using visual and audio features, and to combine content-based similarity search with traditional search methods. Our preliminary experience and evaluation shows that content-based similarity search is easy to use and can achieve 0.79 average precision on our simple benchmark. The system is constructed using Ferret toolkit and its memory footprint for metadata is quite small, requiring about 1.4Gbytes for one year of continuous archived video data.

Patterns and Sequences: Interactive Exploration of Clickstreams to Understand Common Visitor Paths

Modern web clickstream data consists of long, high-dimensional sequences of multivariate events, making it difficult to analyze. Following the overarching principle that the visual interface should provide information about the dataset at multiple levels of granularity and allow users to easily navigate across these levels, we identify four levels of granularity in clickstream analysis: patterns, segments, sequences and events. We present an analytic pipeline consisting of three stages: pattern mining, pattern pruning and coordinated exploration between patterns and sequences. Based on this approach, we discuss properties of maximal sequential patterns, propose methods to reduce the number of patterns and describe design considerations for visualizing the extracted sequential patterns and the corresponding raw sequences. We demonstrate the viability of our approach through an analysis scenario and discuss the strengths and limitations of the methods based on user feedback.

Deep Classifiers from Image Tags in the Wild

This paper proposes direct learning of image classification from image tags in the wild, without filtering. Each wild tag is supplied by the user who shared the image online. Enormous numbers of these tags are freely available, and they give insight about the image categories important to users and to image classification. Our main contribution is an analysis of the Flickr 100 Million Image dataset, including several useful observations about the statistics of these tags. We introduce a large-scale robust classification algorithm, in order to handle the inherent noise in these tags, and a calibration procedure to better predict objective annotations. We show that freely available, wild tag can obtain similar or superior results to large databases of costly manual annotations.

Poisson-uniform nonnegative matrix factorization

Probabilistic models of audio spectrograms used in audio source separation often rely on Poisson or multinomial noise models corresponding to the generalized Kullback-Leibler (GKL) divergence popular in methods using Nonnegative Matrix Factorization (NMF). This noise model works well in practice, but it is difficult to justify since these distributions are technically only applicable to discrete counts data. This issue is particularly problematic in hierarchical and non-parametric Bayesian models where estimates of uncertainty depend strongly on the likelihood model. In this paper, we present a hierarchical Bayesian model that retains the flavor of the Poisson likelihood model but yields a coherent generative process for continuous spectrogram data. This model allows for more principled, accurate, and effective Bayesian inference in probabilistic NMF models based on GKL.

Fast and easy crowdsourced perceptual audio evaluation

Automated objective methods of audio evaluation are fast, cheap, and require little effort by the investigator. However, objective evaluation methods do not exist for the output of all audio processing algorithms, often have output that correlates poorly with human quality assessments, and require ground truth data in their calculation. Subjective human ratings of audio quality are the gold standard for many tasks, but are expensive, slow, and require a great deal of effort to recruit subjects and run listening tests. Moving listening tests from the lab to the micro-task labor market of Amazon Mechanical Turk speeds data collection and reduces investigator effort. However, it also reduces the amount of control investigators have over the testing environment, adding new variability and potential biases to the data. In this work, we compare multiple stimulus listening tests performed in a lab environment to multiple stimulus listening tests performed in web environment on a population drawn from Mechanical Turk.

Exploiting long-term temporal dependencies in NMF using recurrent neural networks with application to source separation

This paper seeks to exploit high-level temporal information during feature extraction from audio signals via non-negative matrix factorization. Contrary to existing approaches that impose local temporal constraints, we train powerful recurrent neural network models to capture long-term temporal dependencies and event co-occurrence in the data. This gives our method the ability to “fill in the blanks” in a smart way during feature extraction from complex audio mixtures, an ability very useful for a number of audio applications. We apply these ideas to source separation problems.

Speech dereverberation using a learned speech model

We present a general single-channel speech dereverberation method based on an explicit generative model of reverberant and noisy speech. To regularize the model, we use a pre-learned speech model of clean and dry speech as a prior and perform posterior inference over the latent clean speech. The reverberation kernel and additive noise are estimated under the maximum-likelihood framework. Our model assumes no prior knowledge about specific speakers or rooms, and consequently our method can automatically adapt to various reverberant and noisy conditions. We evaluate the proposed model with both simulated data and real recordings from the REVERB Challenge1 in the task of speech enhancement and obtain results comparable to or better than the state-of-the-art.

Real-time feature-based synthesis for live musical performance

A crucial set of decisions in digital musical instrument design deals with choosing mappings between parameters controlled by the performer and the synthesis algorithms that actually generate sound. Feature-based synthesis offers a way to parameterize audio synthesis in terms of the quantifiable perceptual characteristics, or features, the performer wishes the sound to take on. Techniques for accomplishing such mappings and enabling feature-based synthesis to be performed in real time are discussed. An example is given of how a real-time performance system might be designed to take advantage of feature-based synthesiss ability to provide perceptually meaningful control over a large number of synthesis parameters.

Speech decoloration based on the product-of-filters model

We present a single-channel speech decoloration method based on a recently proposed generative product-of-filters (PoF) model. We take a spectral approach and attempt to learn the magnitude response of the actual coloration filter, given only the degraded speech signal. Experiments on synthetic data demonstrate that the proposed method effectively captures both coarse and fine structure of the coloration filter. On real recordings, we find that simply subtracting the learned coloration filter from the log-spectra yields promising decoloration results.