Diemo Schwarz

Corpus-Based Concatenative Synthesis

Corpus-based concatenative methods for musical sound synthesis have attracted much attention recently. They make use of a variety of sound snippets in a database to assemble a desired sound or phrase according to a target specification given in sound descriptors or by an example sound. With ever-larger sound databases easily available, together with a pertinent description of their contents, they are increasingly used for composition, high-level instrument synthesis, and interactive exploration of a sound corpus. This article gives an overview of the components needed for corpus-based concatenative synthesis and details of some realizations. Signal processing methods are crucial for all parts of analysis, (segmentation; and descriptor analysis), for synthesis, and can intervene in the selection part, e.g., for spectral matching

Training Ircam's score follower [audio to musical score alignment system]

This paper describes our attempt to make the hidden Markov model (HMM) score following system, developed at Ircam, sensible to past experiences in order to obtain better audio to score real-time alignment for musical applications. A new observation modeling based on Gaussian mixture models is developed which is trainable using a learning algorithm we would call automatic discriminative training. The novelty of this system lies in the fact that this method, unlike classical methods for HMM training, is not concerned with modeling the music signal but with correctly choosing the sequence of music events that was performed. Besides obtaining better alignment, the new systems parameters are controllable in a physical manner and the training algorithm learns different styles of music performance as discussed.

Spectral Envelopes and Additive + Residual Analysis/Synthesis

The subject of this chapter is the estimation, representation, modification and use of spectral envelopes in the context of sinusoidal additive+residual analysis–synthesis. A spectral envelope is a function giving amplitude versus frequency, which is the envelope of a short-time spectrum. Even though the term spectral envelope is commonly used for the envelope of the magnitude of the STS only, we also consider envelopes for the phase of the STS, and even for the frequency of nearly-harmonic partials as a function of their harmonic number. One of the main features of spectral envelopes is that several important properties of sounds are captured in a simple and powerful representation. The concept of spectral envelope is connected with the production models (signal models and physical models) of musical instruments as well as with the perception of musical sounds. Spectral envelopes also offer a simple and concise representation of important sound properties which largely ease the control of synthesis models for musical applications. From the analysis viewpoint, the interest of spectral envelopes relies on the existence of many envelope estimation techniques. Spectral envelopes can be coded in one of several representations which differ by the memory space and the computation power they require. In his chapter, transcoding and manipulation of spectral envelopes are explained, as well as the use of spectral envelopes for the synthesis of sinusoidal and of non-sinusoidal parts of the signal. Some examples of spectral envelope applications for additive+residual analysis and synthesis, in various contexts and software systems, are given. Finally, a few conclusions and perspectives for future development and research concerning spectral envelopes in musical sound signal synthesis are proposed.

Technologyu and Paradigms to Support the Learning of Music Performance

This article gives an overview over the support technology for the learning of musical instrument performance developed and assembled for the I-MAESTRO project and describes some of the developed components in further detail. The underlying paradigms related to the process of music teaching and learning as well as to the processing and representation of data captured from musical instrument performers are mentioned.

Plumage: design d'une interface 3D pour le parcours d'échantillons sonores granularisés

Plumage is an interface for interactive 3D audio/graphic scene browsing and design. The interface relies on the notion of tape heads in a sonic and graphic 3D space made of feathers associated with sound micro-samples. The spatial layout of the feathers is defined by sound parameters of the associated samples. The musical play is the outcome of a continuous and interactive navigation in the sound space controlled by direct manipulation of tape head trajectories. This work is based on the design of interactions. A simple and efficient audio graphic navigation is designed through the combination and synchronization of elementer sound object triggerings.

Collective Sound Checks: Exploring Intertwined Sonic and Social Affordances of Mobile Web Applications

We present the Collective Sound Checks, an exploration of user scenarios based on mobile web applications featuring motion-controlled sound that enable groups of people to engage in spontaneous collaborative sound and music performances. These new forms of musical expression strongly shift the focus of design from human-computer interactions towards the emergence of computer mediated interactions between players based on sonic and social affordances of ubiquitous technologies. At this early stage, our work focuses on experimenting with different user scenarios while observing the relationships between different interactions and affordances.

Dirty tangible interfaces: expressive control of computers with true grit

Dirty Tangible Interfaces (DIRTI) are a new concept in interface design that forgoes the dogma of repeatability in favor of a richer and more complex experience, constantly evolving, never reversible, and infinitely modifiable. We built a prototype interface realizing the DIRTI principles based on low-cost commodity hardware and kitchenware: A video camera tracks a granular or liquid interaction material placed in a glass dish. The 3D relief estimated from the images, and the dynamic changes applied to it by the user(s), are used to control two applications: For 3D scene authoring, the relief is directly translated into a terrain, allowing fast and intuitive map editing. For expressive audio-graphic music performance, both the relief and real-time changes are interpreted as activation profiles to drive corpus-based concatenative sound synthesis, allowing one or more musicians to mold sonic landscapes and to plow through them in an inherently collaborative, expressive, and dynamic experience.

Interactive Sound Texture Synthesis Through Semi-Automatic User Annotations

We present a way to make environmental recordings controllable again by the use of continuous annotations of the high-level semantic parameter one wishes to control, e.g. wind strength or crowd excitation level. A partial annotation can be propagated to cover the entire recording via cross-modal analysis between gesture and sound by canonical time warping (CTW). The annotations serve as a descriptor for lookup in corpus-based concatenative synthesis in order to invert the sound/annotation relationship. The workflow has been evaluated by a preliminary subject test and results on canonical correlation analysis (CCA) show high consistency between annotations and a small set of audio descriptors being well correlated with them. An experiment of the propagation of annotations shows the superior performance of CTW over CCA with as little as 20 s of annotated material.

Spatial audio: graphic modeling for X3D

In audio-graphic scenes, visual and audio modalities are synchronized in time and space, and their behaviour is determined by a common process. We present here a novel way of modeling audio-graphic content for interactive 3D scenes with the concept of sound processes and their activation through 2D or 3D profiles. Many 3D applications today support both graphical and audio effects to provide a more realistic user experience; however a common model and interchange format for interactive audio-graphic scenes is still lacking. X3D is one of the most promising formats for 3D scene representation. It is extensible and supports simple spatial audio representation and almost all basic and advanced 3D computer graphics techniques. We therefore propose an extension of the X3D standard to represent the sound process and activation profile model for providing a rich audio-graphic description in X3D.

Composing Morphology: Concatenative Synthesis as an Intuitive Medium for Prescribing Sound in Time

This article presents AudioGuide, an innovative application for sound synthesis which aims to heighten compositional control of morphology in electronic music. We begin with a discussion of the challenges of managing detail when composing with computers, emphasizing the need for more tools which help the composer address the intricacies of sonic evolution. AudioGuides approach—using a soundfile as a method for specifying morphological shape—provides a simple yet exacting medium for representing temporal ideas. Using the spectral structure of a soundfile, AudioGuide organizes a user-defined collection of pre-recorded sounds to create a similar morphological contour. Our matching strategy accounts for the spectral content, temporal evolution, and superimposition of sonic elements. We provide two examples which illuminate the capabilities of the algorithm from within a musical context.