Masashi Morimoto

Visual pattern discovery using web images

In this paper, a novel approach for discovering visual patterns associated with semantic concepts using web image resources is proposed.This approach can be used to improve the performance in image clustering and retrieval, image annotation, and other applications such as object recognition. Exploring the rich information in web images that represent semantic concepts as both visual content and text information, this research attempts to effectively learn intrinsic patterns related to semantic concepts. Because the quality of learning algorithms is strongly related to the selection of positive and negative samples, positive samples are first selected effectively, then negative samples are determined reliably based on the selected positive samples. Finally, a good quality visual model associated with a semantic concept is built through an unsupervised learning process. The proposed scheme is completely automatic,needing no human intervention,and is robust and reliable for generic images. Experimental results demonstrate the effectiveness of the proposed approach.

SoundCompass: a practical query-by-humming system; normalization of scalable and shiftable time-series data and effective subsequence generation

This paper describes our practical query-by-humming system, SoundCompass, which is being used as a karaoke song selection system in Japan. First, we describe the fundamental techniques employed by SoundCompass such as normalization in a time-wise sense of music data, time-scalable and tone-shiftable time-series data, and making subsequences for efficient matching. Second, we describe techniques to make effective feature vectors based on real music data and do matching with them to develop accurate query-by-humming. Third, we share valuable knowledge that has been obtained through months of practical use of Sound Compass. Fourth, we describe the latest version of the SoundCompass system that incorporates these new techniques and knowledge, as well as describe quantitative evaluations that prove the practicality of SoundCompass. The new system provides flexible and accurate similarity retrieval based on k-nearest neighbor searches with multi-dimensional spatial indices structured with multi-dimensional features vectors.

Real-time background music monitoring based on content-based retrieval

In this paper, we describe music monitoring in TV broadcasting based on content-based retrieval. A part of audio signals is sequentially extracted from TV broadcasting as a retrieval key, and a music DB that stores a great number of musical pieces is retrieved by this key based on content-based retrieval, and a musical piece is identified sequentially. In this way, we are able to carry out music monitoring. There are three necessary requirements important for realization of the music monitoring. They are robustness against non-stationary noise, real-time processing of large-scale music DB retrieval, and high granularity of the retrieval key. As a method of realizing robustness against non-stationary noise, we propose a partially similar retrieval method which improves retrieval accuracy by using the moment in which no superfluous noise is produced during the existence of non-stationary noise. In order to realize real-time processing of large-scale music DB retrieval, we adopt a coarse-to-fine strategy, and propose a spectral peaks hashing method which performs high-speed refining by using hashing. To calculate a hash value in this hashing, frequency channel numbers of the spectral peaks are used. In order to realize high granularity of the retrieval key, it is necessary to solve the problem of retrieval accuracy degradation associated with heightening the granularity. To improve this accuracy, we propose a detection-by-continuity method which uses music continuity. Moreover, by using music continuity to correct the starting point and the terminal point of a musical piece in TV broadcasting, the retrieval accuracy is improved further. In order to evaluate the effectiveness of the proposed methods, we performed experiments using a music DB which stores over 28,000 musical pieces (over 1800 hours) and TV broadcasting audio signals containing music and background music (BGM). The granularity of the retrieval key was set at about 0.5 seconds. Through these experiments, We verified that music monitoring was possible for over 90% of the total time of music and BGM used in TV broadcasting, and that real-time processing was possible.

A smart TV viewing and Web access interface based on video indexing techniques

We propose JoyTV, an advanced TV viewing PC software application. It automatically detects scene changes and music, and recognizes superimposed texts in TV broadcasts. It also links TV contents and Web sites using image features. JoyTV provides users with a smart TV viewing interface based on these video indexes.

Sampling of Web Images with Dictionary Coherence for Cross-Domain Concept Detection

Due to the existence of cross-domain incoherence resulting from the mismatch of data distributions, how to select sufficient positive training samples from scattered and diffused web resources is a challenging problem in the training of effective concept detectors. In this paper, we propose a novel sampling approach to select coherent positive samples from web images for further concept learning based on the degree of image coherence with a given concept. We propose to measure the coherence in terms of how dictionary atoms are shared since shared atoms represent common features with regard to a given concept and are robust to occlusion and corruption. Thus, two kinds of dictionaries are learned through online dictionary learning methods: one is the concept dictionary learned from key-point features of all the positive training samples while the other is the image dictionary learned from those of web images. Intuitively, the coherence degree is then calculated by the Frobenius norm of the product matrix of the two dictionaries. Experimental results show that the proposed approach can achieve constant overall improvement despite cross-domain incoherence.

An Error-Tolerant Video Retrieval Method Based on the Shot Composition Sequence in a Scene

This paper presents an error-tolerant video retrieval method based on the shot composition sequence in a scene. Conventional video players in the home can not access interesting scenes directly because they offer only play, fast-forward, and rewind. What is needed is an easy-to-use video scene player that can directly access scenes important to the user. This paper presents an error-tolerant video retrieval method that assesses the shot composition sequence in each scene. Our method has three characteristics: (1) easy query generation since the regular interface is used for query generation; (2) similar video scene retrieval based on the sequence of symbols created by shot composition clustering to effectively identify scenes similar to the query; (3) fast similarity measurement based on sequences of symbols. An experiment shows that the proposed system can retrieve video scenes similar to the query quickly as well as allowing semantic retrieval to be constructed in a non-supervised way.