Julio J. Carabias-Orti

Musical Instrument Sound Multi-Excitation Model for Non-Negative Spectrogram Factorization

This paper presents theoretical and experimental results about constrained non-negative matrix factorization (NMF) to model the excitation of the musical instruments. These excitations represent vibrating objects, while the filter represents the resonance structure of the instrument, which colors the produced sound. We propose to model the excitations as the weighted sum of harmonically constrained basis functions, whose parameters are tied across different pitches of an instrument. An NMF-based framework is used to learn the model parameters. We assume that the excitations of a well-tempered instrument should possess an identifiable characteristic structure whereas the conditions of the music scene might produce variations in the filter. In order to test the reliability of our proposal, we evaluate our method for a music transcription task in two scenarios. On the first one, comparison with state-of-the-art methods has been performed over a dataset of piano recordings obtaining more accurate results than other NMF-based algorithms. On the second one, two woodwind instrument databases have been used to demonstrate the benefits of our model in comparison with previous excitation-filter model approaches.

Constrained non-negative sparse coding using learnt instrument templates for realtime music transcription

Abstract In this paper, we present a realtime signal decomposition method with single-pitch and harmonicity constrains using instrument specific information. Although the proposed method is designed for monophonic music transcription, it can be used as a candidate selection technique in combination with other realtime transcription methods to address polyphonic signals. The harmonicity constraint is particularly beneficial for automatic transcription because, in this way, each basis can define a single pitch. Furthermore, restricting the model to have a single-nonzero gain at each frame has been shown to be a very suitable constraint when dealing with monophonic signals. In our method, both harmonicity and single-nonzero gain constraints are enforced in a deterministic manner. A realtime factorization procedure based on Non-negative sparse coding (NNSC) with Beta-divergence and fixed basis functions is proposed. In this paper, the basis functions are learned using a supervised process to obtain spectral patterns for different musical instruments. The proposed method has been tested for music transcription of both monophonic and polyphonic signals and has been compared with other state-of-the-art transcription methods, and in these tests, the proposed method has obtained satisfactory results in terms of accuracy and runtime.

Music Scene-Adaptive Harmonic Dictionary for Unsupervised Note-Event Detection

Harmonic decompositions are a powerful tool dealing with polyphonic music signals in some potential applications such as music visualization, music transcription and instrument recognition. The usefulness of a harmonic decomposition relies on the design of a proper harmonic dictionary. Music scene-adaptive harmonic atoms have been used with this purpose. These atoms are adapted to the musical instruments and to the music scene, including aspects related with the venue, musician, and other relevant acoustic properties. In this paper, an unsupervised process to obtain music scene-adaptive spectral patterns for each MIDI-note is proposed. Furthermore, the obtained harmonic dictionary is applied to note-event detection with matching pursuits. In the case of a music database that only consists of one-instrument signals, promising results (high accuracy and low error rate) have been achieved for note-event detection.

Note-event Detection in Polyphonic Musical Signals based on Harmonic Matching Pursuit and Spectral Smoothness

Abstract Harmonic Matching Pursuit (HMP) is an interesting signal processing tool to detect simultaneous notes in musical audio signals. HMP decomposes an audio signal into harmonic atoms, a suitable choice due to their strong harmonic content. However, HMP provides an inaccurate decomposition when notes with a rational frequency relation are simultaneously played (the overlapping partial problem). An example of such a situation is the common occurrence in Western music of consonant intervals, i.e. perfect fifth (3:2), perfect fourth (4:3) or major third (5:4). For these cases, HMP often obtains harmonic atoms whose fundamental frequencies are the greatest common divisor of the active fundamental frequencies. To solve this problem, we propose a processing algorithm that works over the harmonic atoms obtained by HMP. This algorithm is based on maximizing the smoothness of the spectral envelope for each harmonic atom provided by the HMP decomposition. Results show that our proposal achieves a good performance. Competitive accuracy and error measures in a musical transcription application are reported in relation to the state of the art transcription systems.

Nonnegative signal factorization with learnt instrument models for sound source separation in close-microphone recordings

Close-microphone techniques are extensively employed in many live music recordings, allowing for interference rejection and reducing the amount of reverberation in the resulting instrument tracks. However, despite the use of directional microphones, the recorded tracks are not completely free from source interference, a problem which is commonly known as microphone leakage. While source separation methods are potentially a solution to this problem, few approaches take into account the huge amount of prior information available in this scenario. In fact, besides the special properties of close-microphone tracks, the knowledge on the number and type of instruments making up the mixture can also be successfully exploited for improved separation performance. In this paper, a nonnegative matrix factorization (NMF) method making use of all the above information is proposed. To this end, a set of instrument models are learnt from a training database and incorporated into a multichannel extension of the NMF algorithm. Several options to initialize the algorithm are suggested, exploring their performance in multiple music tracks and comparing the results to other state-of-the-art approaches.

Multiple instrument mixtures source separation evaluation using instrument-dependent NMF models

This work makes use of instrument-dependent models to separate the different sources of multiple instrument mixtures. Three different models are applied: (a) basic spectral model with harmonic constraint, (b) source-filter model with harmonic-comb excitation and (c) source-filter model with multi-excitation per instrument. The parameters of the models are optimized by an augmented NMF algorithm and learnt in a training stage. The models are presented in [1], here the experimental setting for the application to source separation is explained. The instrument-dependent NMF models are first trained and then a test stage is performed. A comparison with other state-of-the-art software is presented. Results show that source-filter model with multi-excitation per instrument outperforms the other compared models.

Online Score-Informed Source Separation with Adaptive Instrument Models

In this paper, an online score-informed source separation system is proposed under the Non-negative Matrix Factorization (NMF) framework, using parametric instrument models. Each instrument is modelled using a multi-excitation source-filter model, which provides the flexibility to model different instruments. The instrument models are initially learned on training excerpts of the same kinds of instruments, and are then adapted, during the separation, to the specific instruments used in the audio being separated. The model adaptation method needs to access the musical score content for each instrument, which is provided by an online audio-score alignment method. Source separation is improved by adapting the instrument models using score alignment. Experiments are performed to evaluate the proposed system and its individual components. Results show that it outperforms a state-of-the-art comparison method.

Tempo Driven Audio-to-Score Alignment Using Spectral Decomposition and Online Dynamic Time Warping

In this article, we present an online score following framework designed to deal with automatic accompaniment. The proposed framework is based on spectral factorization and online Dynamic Time Warping (DTW) and has two separated stages: preprocessing and alignment. In the first one, we convert the score into a reference audio signal using a MIDI synthesizer software and we analyze the provided information in order to obtain the spectral patterns (i.e., basis functions) associated to each score unit. In this work, a score unit represents the occurrence of concurrent or isolated notes in the score. These spectral patterns are learned from the synthetic MIDI signal using a method based on Non-negative Matrix Factorization (NMF) with Beta-divergence, where the gains are initialized as the ground-truth transcription inferred from the MIDI. On the second stage, a non-iterative signal decomposition method with fixed spectral patterns per score unit is used over the magnitude spectrogram of the input signal resulting in a distortion matrix that can be interpreted as the cost of the matching for each score unit at each frame. Finally, the relation between the performance and the musical score times is obtained using a strategy based on online DTW, where the optimal path is biased by the speed of interpretation. Our system has been evaluated and compared to other systems, yielding reliable results and performance.

Score-Informed Source Separation for Multichannel Orchestral Recordings

This paper proposes a system for score-informed audio source separation for multichannel orchestral recordings. The orchestral music repertoire relies on the existence of scores. Thus, a reliable separation requires a good alignment of the score with the audio of the performance. To that extent, automatic score alignment methods are reliable when allowing a tolerance window around the actual onset and offset. Moreover, several factors increase the difficulty of our task: a high reverberant image, large ensembles having rich polyphony, and a large variety of instruments recorded within a distant-microphone setup. To solve these problems, we design context-specific methods such as the refinement of score-following output in order to obtain a more precise alignment. Moreover, we extend a close-microphone separation framework to deal with the distant-microphone orchestral recordings. Then, we propose the first open evaluation dataset in this musical context, including annotations of the notes played by multiple instruments from an orchestral ensemble. The evaluation aims at analyzing the interactions of important parts of the separation framework on the quality of separation. Results show that we are able to align the original score with the audio of the performance and separate the sources corresponding to the instrument sections.

Constrained non-negative matrix factorization for score-informed piano music restoration

In this work, we propose a constrained non-negative matrix factorization method for the audio restoration of piano music using information from the score. In the first stage (instrument training), spectral patterns for the target source (piano) are learned from a dataset of isolated piano notes. The model for the piano is constrained to be harmonic because, in this way, each pattern can define a single pitch. In the second stage (noise training), spectral patterns for the undesired source (noise) are learned from the most common types of vinyl noises. To obtain a representative model for the vinyl noise, a cross-correlation-based constraint that minimizes the cross-talk between different noise components is used. In the final stage (separation), we use the trained instrument and noise models in an NMF framework to extract the clean audio signal from undesired non-stationary noise. To improve the separation results, we propose a novel score-based constraint to avoid activations of notes or combinations that are not present in the original score. The proposed approach has been evaluated and compared with commercial audio restoration softwares, obtaining competitive results.