Ichiro Fujinaga

Optical music recognition: state-of-the-art and open issues

For centuries, music has been shared and remembered by two traditions: aural transmission and in the form of written documents normally called musical scores. Many of these scores exist in the form of unpublished manuscripts and hence they are in danger of being lost through the normal ravages of time. To preserve the music some form of typesetting or, ideally, a computer system that can automatically decode the symbolic images and create new scores is required. Programs analogous to optical character recognition systems called optical music recognition (OMR) systems have been under intensive development for many years. However, the results to date are far from ideal. Each of the proposed methods emphasizes different properties and therefore makes it difficult to effectively evaluate its competitive advantages. This article provides an overview of the literature concerning the automatic analysis of images of printed and handwritten musical scores. For self-containment and for the benefit of the reader, an introduction to OMR processing systems precedes the literature overview. The following study presents a reference scheme for any researcher wanting to compare new OMR algorithms against well-known ones.

A Comparative Study of Staff Removal Algorithms

This paper presents a quantitative comparison of different algorithms for the removal of stafflines from music images. It contains a survey of previously proposed algorithms and suggests a new skeletonization-based approach. We define three different error metrics, compare the algorithms with respect to these metrics, and measure their robustness with respect to certain image defects. Our test images are computer-generated scores on which we apply various image deformations typically found in real-world data. In addition to modern western music notation, our test set also includes historic music notation such as mensural notation and lute tablature. Our general approach and evaluation methodology is not specific to staff removal but applicable to other segmentation problems as well.

A Comparison of Approaches to Timbre Descriptors in Music Information Retrieval and Music Psychology

A curious divide characterizes the usage of audio descriptors for timbre research in music information research (MIR) and music psychology. While MIR uses a multitude of audio descriptors for tasks such as automatic instrument classification, only a highly constrained set is used to describe the physical correlates of timbre perception in parts of music psychology. We argue that this gap is not coincidental and results from the differences in the two fields’ methodologies, their epistemic groundwork, and research goals. This paper lays out perspectives on the emergence of the divide and reviews studies in both fields with regards to divergences in research methods and goals. We discuss new representations for spectro-temporal modulations in MIR and psychology, and compare approaches to spectral envelope description in depth. Finally, we will propose that the interdisciplinary discourse on the computational modelling of music requires negotiations about the roles of scientific evaluation criteria.

Using the Gamera framework for the recognition of cultural heritage materials

This paper presents a new toolkit for the creation of customized structured document recognition applications by domain experts. This open-source system, called Gamera, allows a user, with particular knowledge of the documents to be recognized, to combine image processing and recognition tools in an easy-to-use, interactive, graphical scripting environment. Gamera is one of the key technology components in a proposed international project for the digitization of diverse types of humanities documents.

Improving automatic music classification performance by extracting features from different types of data

This paper discusses two sets of automatic musical genre classification experiments. Promising research directions are then proposed based on the results of these experiments. The first set of experiments was designed to examine the utility of combining features extracted from separate and independent audio, symbolic and cultural sources of musical information. The results from this set of experiments indicate that combining feature types can indeed substantively improve classification accuracy as well as reduce the seriousness of those misclassifications that do occur. The second set of experiments examined which high-level features were most important in successfully classifying symbolic data. It was found that features associated with instrumentation were particularly effective. The paper also presents the jMIR toolset, which was used to carry out these experiments and which is particularly well suited to combining information extracted from different types of data sources. jMIR is a free and open-source software suite designed for applications related to automatic music classification of various kinds.

Goal-directed evaluation for the improvement of optical music recognition on early music prints

Optical music recognition (OMR) systems are promising tools for the creation of searchable digital music libraries. Using an adaptive OMR system for early music prints based on hidden Markov models, we leverage an edit distance evaluation metric to improve recognition accuracy. Baseline results are computed with new labeled training and test sets drawn from a diverse group of prints. We present two experiments based on this evaluation technique. The first resulted in a significant improvement to the feature extraction function for these images. The second is a goal-directed comparison of several popular adaptive binarization algorithms, which are often evaluated only subjectively. Accuracy increased by as much as 55% for some pages, and the experiments suggest several avenues for further research.

Perception of touch quality in piano tones

Both timbre and dynamics of isolated piano tones are determined exclusively by the speed with which the hammer hits the strings. This physical view has been challenged by pianists who emphasize the importance of the way the keyboard is touched. This article presents empirical evidence from two perception experiments showing that touch-dependent sound components make sounds with identical hammer velocities but produced with different touch forms clearly distinguishable. The first experiment focused on finger-key sounds: musicians could identify pressed and struck touches. When the finger-key sounds were removed from the sounds, the effect vanished, suggesting that these sounds were the primary identification cue. The second experiment looked at key-keyframe sounds that occur when the key reaches key-bottom. Key-bottom impact was identified from key motion measured by a computer-controlled piano. Musicians were able to discriminate between piano tones that contain a key-bottom sound from those that do not. However, this effect might be attributable to sounds associated with the mechanical components of the piano action. In addition to the demonstrated acoustical effects of different touch forms, visual and tactile modalities may play important roles during piano performance that influence the production and perception of musical expression on the piano.

Document Recognition for a Million Books

As initiatives such as Google Book Search (http://books.google.com/) and the Open Content Alliance (http://www.opencontentalliance.org/) advance efforts to digitize millions of books, there is great potential to make available vast amounts of information. To truly unlock this knowledge, however, it will be necessary to process the resulting digital page images to recognize important content, including both the semantic and structural aspects. Given the vast diversity of fonts, symbols, tables, languages and a host of other elements, it will be necessary to create flexible, modular, scalable document recognition systems. Document recognition involves extracting features from the images and even transcriptions of other documents in order to group diverse content.

Optical Music Interpretation

A system to convert digitized sheet music into a symbolic music representation is presented. A pragmatic approach is used that conceptualizes this primarily two-dimensional structural recognition problem as a one-dimensional one. The transparency of the implementation owes a great deal to its implementation in a dynamic, object-oriented language. This systemis a part of a locally developed end-to-end solution for the conversion of digitized sheet music into symbolic form.

Recommended best practices for digital image capture of musical scores

Like other complex visual articles with small details, musical scores are difficult to capture and present well in digital form. This article presents methods that can be used to reproduce detail and tone from printed scores for creating archival images, based on best practices commonly used by the library community. Capture decisions should be made with a clear idea of the purpose of the imaging project yet be flexible enough to fulfill unanticipated future uses. Options and recommendations for file formats for archival storage, Web delivery and printing of musical materials are discussed.