Agustín Martorell

Hierarchical multi-scale set-class analysis

This work presents a systematic methodology for set-class surface analysis using temporal multi-scale techniques. The method extracts the set-class content of all the possible temporal segments, addressing the representational problems derived from the massive overlapping of segments. A time versus time-scale representation, named class-scape, provides a global hierarchical overview of the class content in the piece, and it serves as a visual index for interactive inspection. Additional data structures summarize the set-class inclusion relations over time and quantify the class and subclass content in pieces or collections, helping to decide about sets of analytical interest. Case studies include the comparative subclass characterization of diatonicism in Victorias masses (in Ionian mode) and Bachs preludes and fugues (in major mode), as well as the structural analysis of Weberns Variations for piano op. 27, under different class-equivalences.

Two-dimensional visual inspection of pitch-space, many time-scales and tonal uncertainty over time

This work explores the representational limitations of toroidal pitch-spaces, when multiple temporal resolutions, tone center ambiguity, and the time dimension are considered for visualization of music pieces. The algorithm estimates key from chroma features, over time at many time-scales, using the key-profile correlation method. All these estimations are projected as tonal centroids within Krumhansl and Kesslers toroidal space of inter-key distances. These centroids, belonging to a toroidal surface, are then mapped to colours by 3-dimensional geometric inscription of the whole pitch-space in the CIELAB colourspace. This mapping provides a visual correlate of pitch-spaces double circularity, approximates perceptual uniformity of colours throughout near regions, and allows for representing key ambiguity. We adapt Sapps keyscapes to summarize tonal centroids in pitch-space at many time-scales over time, in a two-dimensional coloured image. Keyscapes are linked with higher-dimensional tonal representations in a user interface, in order to combine their informative benefits for interactive analysis. By visualizing some specific music examples, we question the potential of continuous toroidal pitch-spaces in supporting long term analytical conclusions and tonal ambiguity description, when assisted by time vs. time-scale representations.

Immersive Orchestras: Audio Processing for Orchestral Music VR Content

This paper combines Audio Signal Processing and Virtual Reality (VR) content to create novel immersive experiences for orchestral music audiences. In VR, the auralization of sound sources of recorded live content remains still a rather unexplored topic. We aim to build a multimodal experience, where visual and audio cues bring a sonic augmentation of the real scene. In the particular scenario of orchestral music content, our goal is to acoustically zoom on a particular instrument when the VR user stares at it. This work aims to improve the learning aspects of music listening, either for education or for personal enrichment. We use audio signal processing to separate different sound sources (instruments) in a acoustic scene (orchestral music recording). Given the signals captured by multiple microphones and the musical score of the piece, our system is able to isolate the different instruments. From the processed separated tracks, we use a binaural rendering technique to emphasize a give instrument. For these experiments we used original content from top European orchestras.

A personality-based adaptive system for visualizing classical music performances

To enhance the experience of listening to classical orchestra music, either in the concert hall or at home, we present a personalized system that integrates three visualization/interaction concepts: Score Follower (points to the current position in the score), Orchestra Layout (illustrates instruments that are currently playing and their dynamics), and Structure Visualization (visualizes structural elements such as themes or motifs). Motivated by previous literature that found evidence for connections between personality and music consumption and preference, we first assessed in a user study to which extent personality traits and music visualization preferences correlate. Measuring preference via pragmatic quality and personality traits according to the Big Five Inventory (BFI) questionnaire, we found substantial interconnections between them. These translate into rules relating certain personality traits (e.g., extraversion or agreeableness) to preference rankings of the visualizations. In the proposed personality-based system, users are grouped into four clusters according to their answers to the most significant personality questions determined in the study. The order of the visualizations for a given user is adapted with respect to the ranking preferred by other users in the same cluster. Evaluation of the system was carried out by a second user study that showed a significantly higher normalized discounted cumulative gain (NDCG) for the personalized system in comparison to a system with randomized order of the visualizations.

Contextual set-class analysis

In this chapter, we review and elaborate a methodology for contextual multi-scale set-class analysis of pieces of music. The proposed method provides a systematic approach to segmentation, description and representation in the analysis of the musical surface. The introduction of a set-class description domain provides a systematic, mid-level, and standard analytical lexicon, which allows for the description of any notated music based on a fixed temperament. The method benefits from representation completeness, a balance between generalization and discrimination of the set-class spaces, and access to hierarchical inclusion relations over time. Three new data structures are derived from the method: class-scapes, class-matrices and class-vectors. A class-scape represents, in a visual way, the set-class content of each possible segment in a piece of music. The class-matrix represents the presence of each possible set class over time, and is invariant to time scale and to several transformations of analytical interest. The class-vector summarizes a piece by quantifying the temporal presence of each possible set class. The balance between dimensionality and informativeness provided by these descriptors is discussed in relation to standard content-based tonal descriptors and music information retrieval applications. The interfacing possibilities of the method are also discussed.