Seungmin Rho

FMF: Query adaptive melody retrieval system

Recent progress of computer and network technologies makes it possible to store and retrieve a large volume of multimedia data in many applications. In such applications, efficient indexing scheme is very important for multimedia retrieval. Depending on the media type, multimedia data shows distinct characteristics and requires different approach to handle. In this paper, we propose a fast melody finder (FMF) that can retrieve melodies fast from audio database based on frequently queried tunes. Those tunes are collected from user queries and incrementally updated into index. Considering empirical user request pattern for multimedia data, those tunes will cover significant portion of user requests. FMF represents all the acoustic and common music notational inputs using a well-known string format such as UDR and LSR and uses string matching techniques to find query results. We implemented a prototype system and report on its performance through various experiments.

MUSEMBLE: A novel music retrieval system with automatic voice query transcription and reformulation

So far, many researches have been done to develop efficient music retrieval systems, and query-by-humming has been considered as one of the most intuitive and effective query methods for music retrieval. For the voice humming to be a reliable query source, elaborate signal processing and acoustic similarity measurement schemes are necessary. On the other hand, recently, there has been an increased interest in query reformulation using relevance feedback with evolutionary techniques such as genetic algorithm for multimedia information retrieval. However, these techniques have not been exploited widely in the field of music retrieval. In this paper, we develop a novel music retrieval system called MUSEMBLE (MUSic enEMBLE) based on two distinct features: (i) A sung or hummed query is automatically transcribed into a sequence of pitch and duration pairs with improved accuracy for music representation. More specifically, we developed two new and unique techniques called WAE (windowed average energy) and dynamic ADF (amplitude-based difference function) onsets for more accurate note segmentation and onset/offset detection in acoustic signal, respectively. The former improved energy-based approaches such as AE by defining small but coherent windows with local and global threshold values. On the other hand, the latter improved the AF (amplitude function) that calculates the summation of the absolute values of signal differences for the clustering energy contour. (ii) A user query is reformulated using user relevance feedback with a genetic algorithm to improve retrieval performance. Even though we have especially focused on humming queries in this paper, MUSEMBLE provides versatile query and browsing interfaces for various kinds of users. We have carried out extensive experiments on the prototype system to evaluate the performance of our voice query transcription and genetic algorithm-based relevance feedback schemes. We demonstrate that our proposed method improves the retrieval accuracy up to 20-40% compared with other popular RF methods. We also show that both WAE and Dynamic ADF methods improve the transcription accuracy up to 95%.

FMF(Fast Melody Finder): A Web-Based Music Retrieval System

As the use of digital music is getting popular, there is an increasing demand for efficient retrieval of music. To do that, an effective music indexing and natural way of querying a music should be incorporated. This paper describes the FMF system that designed to retrieve tunes from a database on the basis of a few notes which are drawn into a musical sheet applet or sung into a microphone. FMF system accepts both acoustic and visual input from the user, transcribes all the acoustic and common music notational inputs into specific strings such as UDR and LSR. Then, It searches an index for tunes that contain the sung pattern, or patterns similar to it. We implemented a web-based retrieval system and report on its performance through various experiments.

Music Information Retrieval Using a GA-based Relevance Feedback

Recently, there has been an increased interest in the query reformulation using relevance feedback with evolutionary techniques such as genetic algorithm for multimedia information retrieval. However, these techniques have still not been exploited widely in the field of music retrieval. In this paper, we propose a novel music retrieval scheme that incorporates user relevance feedback with genetic algorithm to improve retrieval performance and develop a prototype system based on it. Our system also provides interesting easy-to-use graphical user interfaces. For example, users can browse and play query results easily using markers in the music indicating those matched parts for the query. By performing various experiments, we show the effectiveness and efficiency of our proposed scheme.

An Efficient Voice Transcription Scheme for Music Retrieval

In this paper, we propose a new scheme for transcribing sung or hummed queries into a sequence of pitch and duration pairs automatically for efficient music retrieval. More specifically, we present two novel methods called WAE (windowed average energy) and dynamic threshold method for ADF onsets for note segmentation and onset/offset detection in acoustic signal, respectively. The former improves previous energy-based approaches such as AE by defining small but coherent windows with local and global threshold values. The latter also improves the traditional global/local threshold method. By performing various experiments on our prototype music retrieval system, we show the effectiveness of our proposed scheme.

Video scene determination using audiovisual data analysis

While current approaches for segmenting or classifying audiovisual data are mostly focused on visual information, audio information may also play an important role in the content parsing for many applications. We propose a novel scheme for determining video scenes by analyzing both audio and video data. We also describe some preliminary experiments that we have performed and report some of the results in the automatic segmentation and classification of audiovisual data for video indexing and retrieval. Audio segmentation and classification are used not only for extracting useful video indexing information, but also for performing a preprocessing step for other analysis modules.

M-MUSICS: mobile content-based music retrieval system

Accurate voice humming transcription and efficient indexing schemes are essential for a large-scale humming-based music retrieval system. Although many researches have been done to develop such schemes, their performances are not still satisfactory. In our previous works, we proposed (i) a new voice query transcription scheme [4], (ii) a popularity-adaptive indexing structure called FAI [6] for fast retrieval, and (iii) a semi-supervised relevance feedback and query reformulation scheme based on a genetic algorithm [7] in order to improve retrieval efficiency. In this demonstration, we extend our efforts to a mobile environment and develop a prototype mobile music retrieval system called M-MUSICS. Our focus in this implementation includes versatile user interface for easy querying and browsing on a typical mobile device such as PDA phone and satisfactory performance in a wireless mobile environment. We report some of the results.

Adaptive multimedia content delivery in ubiquitous environments

Due to the dramatic technology development in Internet and wireless communication, it becomes possible to access multimedia contents and services using handheld devices. However, most multimedia contents and services on the Web are optimized for desktop computing environment. Even though multimedia contents can represent rich semantics, they are not suitable for handheld devices with limited resources and computing capabilities. Moreover, depending on user preference or usage scenarios, not all the data are relevant or critical to the application. In this paper, we present an adaptation scheme of multimedia contents on the wireless devices for QoS-aware delivery. Especially, we seek to adapt multimedia contents based on available system and network resources, client terminals, user preferences and some other environmental characteristics.

MUSEMBLE: A Music Retrieval System Based on Learning Environment

Query reformulation has been suggested as an effective way to improve retrieval efficiency in text information retrieval and one of the well-known techniques for query reformulation is user relevance feedback. Recently, there has been an increased interest in the query reformulation using relevance feedback with evolutionary techniques such as genetic algorithm for multimedia information retrieval. However, these techniques have still not been exploited widely in the field of music retrieval. In this paper, we propose a novel music retrieval scheme that is based on user relevance feedback with genetic algorithm and evolutionary method with neural network. The former is for reformulating a user query and the latter is for reducing the population size by learning neural network. We implemented a prototype music retrieval system called MUSEMBLE based on this scheme. Experimental results showed that our proposed scheme achieves a good performance.

Voice Query Transcription and Expansion Scheme for Efficient Music Retrieval

In this paper, we present a scheme for efficient humming-based music retrieval. For that purpose, we first describe how to extract a sequence of pitch and duration pairs as its feature information from sung or hummed query accurately and automatically. And then, we propose a novel scheme for reformulating user query to improve retrieval performance. The scheme is based on user relevance feedback with genetic algorithm. We implemented a prototype system based on these scheme and performed various experiments. Experimental result shows that our proposed scheme achieves an impressive performance.