Alexander Schindler

Capturing the Temporal Domain in Echonest Features for Improved Classification Effectiveness

This paper proposes Temporal Echonest Features to harness the information available from the beat-aligned vector sequences of the features provided by The Echo Nest. Rather than aggregating them via simple averaging approaches, the statistics of temporal variations are analyzed and used to represent the audio content. We evaluate the performance on four traditional music genre classification test collections and compare them to state of the art audio descriptors. Experiments reveal, that the exploitation of temporal variability from beat-aligned vector sequences and combinations of different descriptors leads to an improvement of classification accuracy. Comparing the results of Temporal Echonest Features to those of approved conventional audio descriptors used as benchmarks, these approaches perform well, often significantly outperforming their predecessors, and can be effectively used for large scale music genre classification.

Quality assurance for document image collections in digital preservation

Maintenance of digital image libraries requires to frequently asses the quality of the images to engage preservation measures if necessary. We present an approach to image based quality assurance for digital image collections based on local descriptor matching. We use spatially distinctive local keypoints of contrast enhanced images and robust symmetric descriptor matching to calculate affine transformations for image registration. Structural similarity of aligned images is used for quality assessment. The results show, that our approach can efficiently asses the quality of digitized documents including images of blank paper.

An Audio-Visual Approach to Music Genre Classification through Affective Color Features

This paper presents a study on classifying music by affective visual information extracted frommusic videos. The proposed audio-visual approach analyzes genre specific utilization of color. A comprehensive set of color specific image processing features used for affect and emotion recognition derived from psychological experiments or art-theory is evaluated in the visual and multi-modal domain against contemporary audio content descriptors. The evaluation of the presented color features is based on comparative classification experiments on the newly introduced ‘Music Video Dataset’. Results show that a combination of the modalities can improve non-timbral and rhythmic features but show insignificant effects on high performing audio features.

Harnessing Music-Related Visual Stereotypes for Music Information Retrieval

Over decades, music labels have shaped easily identifiable genres to improve recognition value and subsequently market sales of new music acts. Referring to print magazines and later to music television as important distribution channels, the visual representation thus played and still plays a significant role in music marketing. Visual stereotypes developed over decades that enable us to quickly identify referenced music only by sight without listening. Despite the richness of music-related visual information provided by music videos and album covers as well as T-shirts, advertisements, and magazines, research towards harnessing this information to advance existing or approach new problems of music retrieval or recommendation is scarce or missing. In this article, we present our research on visual music computing that aims to extract stereotypical music-related visual information from music videos. To provide comprehensive and reproducible results, we present the Music Video Dataset, a thoroughly assembled suite of datasets with dedicated evaluation tasks that are aligned to current Music Information Retrieval tasks. Based on this dataset, we provide evaluations of conventional low-level image processing and affect-related features to provide an overview of the expressiveness of fundamental visual properties such as color, illumination, and contrasts. Further, we introduce a high-level approach based on visual concept detection to facilitate visual stereotypes. This approach decomposes the semantic content of music video frames into concrete concepts such as vehicles, tools, and so on, defined in a wide visual vocabulary. Concepts are detected using convolutional neural networks and their frequency distributions as semantic descriptions for a music video. Evaluations showed that these descriptions show good performance in predicting the music genre of a video and even outperform audio-content descriptors on cross-genre thematic tags. Further, highly significant performance improvements were observed by augmenting audio-based approaches through the introduced visual approach.

The Europeana Sounds Music Information Retrieval Pilot

This paper describes the realization of a Music Information Retrieval (MIR) pilot for a huge audio corpora of European cultural sound heritage, which was developed as part of the Europeana Sounds project. The demonstrator aimed at evaluating the applicability of technologies deriving from the MIR domain to content provided by various European digital libraries and audio archives. To approach this aim, a query-by-example functionality was implemented using audio-content based similarity search. The development was preceded by an elaborated evaluation of the Europeana Sounds collection to assess appropriate combinations of music content descriptors that are capable to effectively discriminate the various types of audio-content provided within the dataset. The MIR-pilot was evaluated both by using an automatic and a user based evaluation. The results showed that the quality of the implemented query-by-example algorithm is comparable to state-of-the-art music similarity approaches reported in literature.

A Picture is Worth a Thousand Songs: Exploring Visual Aspects of Music

The abstract nature of music makes it intrinsically hard to describe. To alleviate this problem we use well known songs or artists as a reference to describe new music. Music information research has mimicked this behavior by introducing search systems that rely on prototypical songs. Based on similarity models deducing from signal processing or collaborative filtering an according list of songs with similar properties is retrieved. Yet, music is often searched for a specific intention such as music for workout, to focus or for a comfortable dinner with friends. Modeling music similarities based on such criteria is in many cases problematic or visionary. Despite these open research challenges a more user focused question should be raised: Which interface is adequate for describing such intentions? Traditionally queries are either based on text input or seed songs. Both are in many cases inadequate or require extensive interaction or knowledge from the user. Despite the multi-sensory capabilities of humans, we primarily focus on vision. Many intentions for music searches can easily be pictorially envisioned. This paper suggests to pursue a query music by image approach. Yet, extensive research in all disciplinary fields of music research, such as music psychology, musicology and information technologies, is required to identify correlations between the acoustic and the visual domain. This paper elaborates on opportunities and obstacles and proposes ways to approach the stated problems.

An Image Based Approach for Content Analysis in Document Collections

We consider the task of content based analysis and categorization in large-scale historical book scanning projects. Mixed content, deprecated language, noise and unexpected distortions suggest an image based approach. The use of keypoint extractors combined with the bag of features approach is applied to scanned text documents. In order to incorporate spatial information into the bag of features approach we consider three methods of spatial verification. An approach based on comparison of statistical properties of local keypoint properties such as size orientation and scale showed comparable quality in content comparison while being computationally much more efficient. Cluster analysis delivers groups of pages characterized by common properties, especially duplicated page content is detected with high reliability.

Duplicate detection approaches for quality assurance of document image collections

This paper presents an evaluation of different methods for automatic duplicate detection in digitized collections. These approaches are meant to support quality assurance and decision making for long term preservation of digital content in libraries and archives. In this paper we demonstrate advantages and drawbacks of different approaches. Our goal is to select the most efficient method which satisfies the digital preservation requirements for duplicate detection in digital document image collections. Workflows of different complexity were designed in order to demonstrate possible duplicate detection approaches. Assessment of individual approaches is based on workflow simplicity, detection accuracy and acceptable performance, since image processing methods typically require significant computation. Applied image processing methods create expert knowledge that facilitates decision making for long term preservation. We employ AI technologies like expert rules and clustering for inferring explicit knowledge on the content of the digital collection. A statistical analysis of the aggregated information and the qualitative analysis of the aggregated knowledge are presented in the evaluation part of the paper.

Automatic classification of defect page content in scanned document collections

We describe a method for defect detection and classification for collections of digital images of historical book documents. Undistorted text images from various books characterized by strong variation of language, font and layout properties are discriminated from typical errors in digitization processes such as occlusion by an operators hand, visible book edge or image warping artifacts. A bag of local features approach is compared to a global characterization of location, size and orientation properties of detected keypoints. Machine learning is used to discriminate between those classes. Results for different features are compared for the task of discrimination between undistorted text and the major distortion class which is presence of the operators hand, where features based on the bag of local features derived histograms achieved a cross-validation accuracy better than 99 percent on a representative data set. Taking into account up to three classes of distortions still resulted in cross-validation accuracies beyond 90 percent using bag of local features derived visual histograms for classifier input.

An expert system for quality assurance of document image collections

Digital preservation workflows for automatic acquisition of image collections are susceptible to errors and require quality assurance. This paper presents an expert system that supports decision making for page duplicate detection in document image collections. Our goal is to create a reliable inference engine and a solid knowledge base from the output of an image processing tool that detects duplicates based on methods of computer vision. We employ artificial intelligence technologies (i.e. knowledge base, expert rules) to emulate reasoning about the knowledge base similar to a human expert. A statistical analysis of the automatically extracted information from the image comparison tool and the qualitative analysis of the aggregated knowledge are presented.