Nanzhu Jiang

A Robust Fitness Measure for Capturing Repetitions in Music Recordings With Applications to Audio Thumbnailing

The automatic extraction of structural information from music recordings constitutes a central research topic. In this paper, we deal with a subproblem of audio structure analysis called audio thumbnailing with the goal to determine the audio segment that best represents a given music recording. Typically, such a segment has many (approximate) repetitions covering large parts of the recording. As the main technical contribution, we introduce a novel fitness measure that assigns a fitness value to each segment that expresses how much and how well the segment “explains” the repetitive structure of the entire recording. The thumbnail is then defined to be the fitness-maximizing segment. To compute the fitness measure, we describe an optimization scheme that jointly performs two error-prone steps, path extraction and grouping, which are usually performed successively. As a result, our approach is even able to cope with strong musical and acoustic variations that may occur within and across related segments. As a further contribution, we introduce the concept of fitness scape plots that reveal global structural properties of an entire recording. Finally, to show the robustness and practicability of our thumbnailing approach, we present various experiments based on different audio collections that comprise popular music, classical music, and folk song field recordings.

Novel audio features for capturing tempo salience in music recordings

In music compositions, certain parts may be played in an improvisational style with a rather vague notion of tempo, while other parts are characterized by having a clearly perceivable tempo. Based on this observation, we introduce in this paper some novel audio features for capturing tempo-related information. Rather than measuring the specific tempo of a local section of a given recording, our objective is to capture the existence or absence of a notion of tempo, a kind of tempo salience. By a quantitative analysis within an Indian music scenario, we demonstrate that our audio features capture the aspect of tempo salience well, while being independent of continuous fluctuations and local changes in tempo.

Towards efficient audio thumbnailing

Audio thumbnailing, which aims at finding the most representative audio segment of a music recording, is an important task in music information retrieval. In this paper, we show how the computational efficiency of a recently proposed state-of-the-art thumbnailing approach can be improved significantly. The basic idea of the previous approach is to compute for each possible segment a fitness value that expresses repetitiveness and then to define the thumbnail as the fitness-maximizing segment. As a first acceleration strategy, we propose an efficient multi-level sampling strategy to reduce the number of segments the fitness has to be computed for. Second, we obtain further accelerations by suitably adjusting the resolution used in the fitness computation depending on the level of the segment. As a third contribution, we exploit an intrinsic property of the fitness computation that allows us to estimate the fitness for certain segments without any further computation. Our experimental results show that combining these three strategies leads to accelerations by a factor of 20 to 200 depending on the duration of the song while keeping the overall accuracy for the thumbnail estimation.

Estimating double thumbnails for music recordings

Audio thumbnailing, which aims at finding the most representative audio segment of a music recording, is an important task in music information retrieval. In general, the notion of a thumbnail is not well-defined and several musical parts may be good thumbnail candidates. For example, for popular music, both a verse and a refrain section may serve as suitable thumbnail candidates. Instead of considering only one thumbnail, we consider in this paper the problem of finding the two most representative segments that correspond to different musical parts. We denote these two segments as double thumbnails. As our main technical contributions, we propose two approaches for computing double thumbnails, both extending a previously introduced repetition-based thumbnailing procedure. In the first approach, which is straightforward, we simply apply the original thumbnailing procedure two times in an iterative fashion. In the second approach, we introduce a novel method for jointly estimating the two thumbnails within one optimization procedure. Finally, we report on experimental results demonstrating the performances of the two double thumbnailing procedures and indicate directions towards full music structure analysis.