Markus Schedl

A music search engine built upon audio-based and web-based similarity measures

An approach is presented to automatically build a search engine for large-scale music collections that can be queried through natural language. While existing approaches depend on explicit manual annotations and meta-data assigned to the individual audio pieces, we automatically derive descriptions by making use of methods from Web Retrieval and Music Information Retrieval. Based on the ID3 tags of a collection of mp3 files, we retrieve relevant Web pages via Google queries and use the contents of these pages to characterize the music pieces and represent them by term vectors. By incorporating complementary information about acous tic similarity we are able to both reduce the dimensionality of the vector space and improve the performance of retrieval, i.e. the quality of the results. Furthermore, the usage of audio similarity allows us to also characterize audio pieces when there is no associated information found on the Web.

Polyphonic piano note transcription with recurrent neural networks

In this paper a new approach for polyphonic piano note onset transcription is presented. It is based on a recurrent neural network to simultaneously detect the onsets and the pitches of the notes from spectral features. Long Short-Term Memory units are used in a bidirectional neural network to model the context of the notes. The use of a single regression output layer instead of the often used one-versus-all classification approach enables the system to significantly lower the number of erroneous note detections. Evaluation is based on common test sets and shows exceptional temporal precision combined with a significant boost in note transcription performance compared to current state-of-the-art approaches. The system is trained jointly with various synthesized piano instruments and real piano recordings and thus generalizes much better than existing systems.

The neglected user in music information retrieval research

Personalization and context-awareness are highly important topics in research on Intelligent Information Systems. In the fields of Music Information Retrieval (MIR) and Music Recommendation in particular, user-centric algorithms should ideally provide music that perfectly fits each individual listener in each imaginable situation and for each of her information or entertainment needs. Even though preliminary steps towards such systems have recently been presented at the “International Society for Music Information Retrieval Conference” (ISMIR) and at similar venues, this vision is still far away from becoming a reality. In this article, we investigate and discuss literature on the topic of user-centric music retrieval and reflect on why the breakthrough in this field has not been achieved yet. Given the different expertises of the authors, we shed light on why this topic is a particularly challenging one, taking computer science and psychology points of view. Whereas the computer science aspect centers on the problems of user modeling, machine learning, and evaluation, the psychological discussion is mainly concerned with proper experimental design and interpretation of the results of an experiment. We further present our ideas on aspects crucial to consider when elaborating user-aware music retrieval systems.

A survey of music similarity and recommendation from music context data

In this survey article, we give an overview of methods for music similarity estimation and music recommendation based on music context data. Unlike approaches that rely on music content and have been researched for almost two decades, music-context-based (or contextual) approaches to music retrieval are a quite recent field of research within music information retrieval (MIR). Contextual data refers to all music-relevant information that is not included in the audio signal itself. In this article, we focus on contextual aspects of music primarily accessible through web technology. We discuss different sources of context-based data for individual music pieces and for music artists. We summarize various approaches for constructing similarity measures based on the collaborative or cultural knowledge incorporated into these data sources. In particular, we identify and review three main types of context-based similarity approaches: text-retrieval-based approaches (relying on web-texts, tags, or lyrics), co-occurrence-based approaches (relying on playlists, page counts, microblogs, or peer-to-peer-networks), and approaches based on user ratings or listening habits. This article elaborates the characteristics of the presented context-based measures and discusses their strengths as well as their weaknesses.

An innovative three-dimensional user interface for exploring music collections enriched

We present a novel, innovative user interface to music repositories. Given an arbitrary collection of digital music files, our system creates a virtual landscape which allows the user to freely navigate in this collection. This is accomplished by automatically extracting features from the audio signal and training a Self-Organizing Map (SOM) on them to form clusters of similar sounding pieces of music. Subsequently, a Smoothed Data Histogram (SDH) is calculated on the SOM and interpreted as a three-dimensional height profile. This height profile is visualized as a three-dimensional island landscape containing the pieces of music. While moving through the terrain, the closest sounds with respect to the listeners current position can be heard. This is realized by anisotropic auralization using a 5.1 surround sound model. Additionally, we incorporate knowledge extracted automatically from the web to enrich the landscape with semantic information. More precisely, we display words and related images that describe the heard music on the landscape to support the exploration.

Exploring Music Collections in Virtual Landscapes

A user interface to music repositories called nepTune creates a virtual landscape for an arbitrary collection of digital music files, letting users freely navigate the collection. Automatically extracting features from the audio signal and clustering the music pieces accomplish this. The clustering helps generate a 3D island landscape. The rapidly growing research field of music information retrieval is developing the technological foundations for a new generation of more intelligent music devices and services. Researchers are creating algorithms for audio and music analysis, studying methods for retrieving music-related information from the Internet, and investigating scenarios for using music-related information for novel types of computer-based music services. The range of applications for such technologies is broad - from automatic music recommendation services through personalized, adaptive radio stations, to novel types of intelligent, reactive musical devices and environments.

Evaluation in Music Information Retrieval

The field of Music Information Retrieval has always acknowledged the need for rigorous scientific evaluations, and several efforts have set out to develop and provide the infrastructure, technology and methodologies needed to carry out these evaluations. The community has enormously gained from these evaluation forums, but we have reached a point where we are stuck with evaluation frameworks that do not allow us to improve as much and as well as we want. The community recently acknowledged this problem and showed interest in addressing it, though it is not clear what to do to improve the situation. We argue that a good place to start is again the Text IR field. Based on a formalization of the evaluation process, this paper presents a survey of past evaluation work in the context of Text IR, from the point of view of validity, reliability and efficiency of the experiments. We show the problems that our community currently has in terms of evaluation, point to several lines of research to improve it and make various proposals in that line.

Location-Aware Music Artist Recommendation

Current advances in music recommendation underline the importance of multimodal and user-centric approaches in order to transcend limits imposed by methods that solely use audio, web, or collaborative filtering data. We propose several hybrid music recommendation algorithms that combine information on the music content, the music context, and the user context, in particular integrating geospatial notions of similarity. To this end, we use a novel standardized data set of music listening activities inferred from microblogs ( MusicMicro ) and state-of-the-art techniques to extract audio features and contextual web features. The multimodal recommendation approaches are evaluated for the task of music artist recommendation. We show that traditional approaches (in particular, collaborative filtering) benefit from adding a user context component, geolocation in this case.

“Reinventing the Wheel”: A Novel Approach to Music Player Interfaces

We present a novel interface to (portable) music players that benefit from intelligently structured collections of audio files. For structuring, we calculate similarities between every pair of songs and model a travelling salesman problem (TSP) that is solved to obtain a playlist (i.e., the track ordering during playback) where the average distance between consecutive pieces of music is minimal according to the similarity measure. The similarities are determined using both audio signal analysis of the music tracks and Web-based artist profile comparison. Indeed, we show how to enhance the quality of the well-established methods based on audio signal processing with features derived from Web pages of music artists. Using TSP allows for creating circular playlists that can be easily browsed with a wheel as input device. We investigate the usefulness of four different TSP algorithms for this purpose. For evaluating the quality of the generated playlists, we apply a number of quality measures to two real-world music collections. It turns out that the proposed combination of audio and text-based similarity yields better results than the initial approach based on audio data only. We implemented an audio player as Java applet to demonstrate the benefits of our approach. Furthermore, we present the results of a small user study conducted to evaluate the quality of the generated playlists

The CoMIRVA toolkit for visualizing music-related data

We present CoMIRVA, which is an abbreviation for Collection of Music Information Retrieval and Visualization Applications. CoMIRVA is a Java framework and toolkit for information retrieval and visualization. It is licensed under the GNU GPL and can be downloaded from http://www.cp.jku.at/comirva/. At the moment, the main functionalities include music information retrieval, web retrieval, and visualization of the extracted information. In this paper, we focus on the visualization aspects of CoMIRVA. Since many of the information retrieval functions are intended to be applied to problems of the field of music information retrieval (MIR), we demonstrate the functions using data like similarity matrices of music artists gained by analyzing artist-related web pages. CoMIRVA is continuously being extended. Currently, it supports the following visualization techniques: Self-Organizing Map, Smoothed Data Histogram, Circled Bars, Circled Fans, Probabilistic Network, Continuous Similarity Ring, Sunburst, and Music Description Map. Since space is limited, we can only present a selected number of these in this paper. As one key feature of CoMIRVA is its easy extensibility, we further elaborate on how CoMIRVA was used for creating a novel user interface to digital music repositories.