Gérard Assayag

Computer-Assisted Composition at IRCAM: From PatchWork to OpenMusic

In recent years, IRCAM has been exploring and developing software for computer-assisted composition (CAC). These software packages allow composers and musicologists to formalize and experiment with the structures and dynamics of musical languages. The Formes program (Rodet and Cointe 1984), although primarily devoted to the control of sound synthesis, was really a compositional environment with a high-level object-oriented architecture. The Crime environment (Assayag, Castellengo, and Malherbe 1985; Amiot, Assayag, Malherbe, and Riotte 1986) was the first attempt at IRCAM to realize a general CAC environment where the user could define and control abstract musical formalisms. Francis Courtot developed CARLA as an attempt to use a visual programming interface to a Prolog-based logic-programming system (Balaban, Ebcioglu, and Laske 1992). The development of the PatchWork environment, by M. Laurson, J. Duthen, and C. Rueda (Laurson and Duthen 1989; Laurson 1996), was the next stage in the development of CAC programs at IRCAM. The combination of programming simplicity and a highly visual interface in a personal computing concept created an infatuation with PatchWork among European composers with highly diverse musical and aesthetic backgrounds, including Antoine Bonnet, Michel Fano, Brian Ferneyhough, Gerard Grisey, Paavo Heininen, Magnus Lindberg, Claudy Malherbe, Tristan Murail, Kaija Saariaho, and many others. OpenMusic, designed by G. Assayag and C. Agon (Assayag, Agon, Fineberg, and Hanappe 1997; Agon, Assayag, Delerue, and Rueda 1998), is the most recent IRCAM CAC environment. It is a visual interface to CLOS, the Common Lisp Object System (Steele 1990). Aside from being a superset of PatchWork, it opens new territories by allowing the composer to visually design sophisticated musical object classes. It introduces the maquette concept, which enables high-level control of musical material over time, and it revises the PatchWork visual language in a modern way.

Using Factor Oracles for Machine Improvisation

We describe variable markov models we have used for statistical learning of musical sequences, then we present the factor oracle, a data structure proposed by Crochemore & al for string matching. We show the relation between this structure and the previous models and indicate how it can be adapted for learning musical sequences and generating improvisations in a real-time context.

Using machine-learning methods for musical style modeling

The ability to construct a musical theory from examples presents a great intellectual challenge that, if successfully met, could foster a range of new creative applications. Inspired by this challenge, we sought to apply machine-learning methods to the problem of musical style modeling. Our work so far has produced examples of musical generation and applications to a computer-aided composition system. Machine learning consists of deriving a mathematical model, such as a set of stochastic rules, from a set of musical examples. The act of musical composition involves a highly structured mental process. Although it is complex and difficult to formalize, it is clearly far from being a random activity. Our research seeks to capture some of the regularity apparent in the composition process by using statistical and information theoretic tools to analyze musical pieces. The resulting models can be used for inference and prediction and, to a certain extent, to generate new works that imitate the style of the great masters.

OMax brothers: a dynamic yopology of agents for improvization learning

We describe a multi-agent architecture for an improvization oriented musician-machine interaction system that learns in real time from human performers. The improvization kernel is based on sequence modeling and statistical learning. The working system involves a hybrid architecture using two popular composition/perfomance environments, Max and OpenMusic, that are put to work and communicate together, each one handling the process at a different time/memory scale. The system is capable of processing real-time audio/video as well as MIDI. After discussing the general cognitive background of improvization practices, the statistical modeling tools and the concurrent agent architecture are presented. Finally, a prospective Reinforcement Learning scheme for enhancing the systems realism is described.

On the Information Geometry of Audio Streams With Applications to Similarity Computing

This paper proposes methods for information processing of audio streams using methods of information geometry. We lay the theoretical groundwork for a framework allowing the treatment of signal information as information entities, suitable for similarity and symbolic computing on audio signals. The theoretical basis of this paper is based on the information geometry of statistical structures representing audio spectrum features, and specifically through the bijection between the generic families of Bregman divergences and that of exponential distributions. The proposed framework, called Music Information Geometry, allows online segmentation of audio streams to metric balls where each ball represents a quasi-stationary continuous chunk of audio, and discusses methods to qualify and quantify information between entities for similarity computing. We define an information geometry that approximates a similarity metric space, redefine general notions in music information retrieval such as similarity between entities, and address methods for dealing with nonstationarity of audio signals. We demonstrate the framework on two sample applications for online audio structure discovery and audio matching.

Anticipatory Model of Musical Style Imitation Using Collaborative and Competitive Reinforcement Learning

The role of expectationin listening and composing music has drawn much attention in music cognition since about half a century ago. In this paper, we provide a first attempt to model some aspects of musical expectation specifically pertained to short-time and working memories, in an anticipatory framework. In our proposition anticipationis the mental realization of possible predicted actions and their effect on the perception of the world at an instant in time. We demonstrate the model in applications to automatic improvisation and style imitation. The proposed model, based on cognitive foundations of musical expectation, is an active model using reinforcement learning techniques with multiple agents that learn competitively and in collaboration. We show that compared to similar models, this anticipatory framework needs little training data and demonstrates complex musical behavior such as long-term planning and formal shapes as a result of the anticipatory architecture. We provide sample results and discuss further research.

Integrating Constraints and Concurrent Objects in MusicalApplications: A Calculus and its Visual Language

We propose PiCO, a calculus integrating concurrent objects and constraints, as a base for music composition tools. In contrast with calculi such as NiehrenMueller:Free, milner.parrow.ea:calculus-mobile or TyCO vasconcelos:typed-concurrent, both constraints and objects are primitive notions in PiCO. In PiCO a base object model is extended with constraints by orthogonally adding the notion of constraint system found in the ρ-calculus OzCalculus. Concurrent processes make use of a constraint store to synchronize communications either via the ask and tell operations of the constraint model or the standard message-passing mechanism of the object model. A message delegation mechanism built into the calculus allows encoding of general forms of inheritance. This paper includes encodings in PiCO of the concepts of class and sub-class. These allow us to represent complex partially defined objects such as musical structures in a compact way. We illustrate the transparent interaction of constraints and objects by a musical example involving harmonic and temporal relations. The relationship between Cordial, a visual language for music composition applications, and its underlying model PiCO is described.

Audio Oracle Analysis of Musical Information Rate

This paper presents a method for analysis of changes in information contents in music based on an audio representation called Audio Oracle (AO). Using compression properties of AO we estimate the amount of information that passes between the past and the present at every instance in a musical signal. This formulation extends the notion of Information Rate (IR) to individual sequences and allows an optimal estimation of the AO threshold parameter. We show that changes in IR correspond to significant musical structures such as sections in a sonata form. Relation to musical perception and applications for composition and improvisation are discussed in the paper.

Iscore: a system for writing interaction

In this article, we present the development of research carried out to design a system of interactive scores for composition and execution, based on temporal constraints called Iscore. This system has been designed in order to allow musicians to interpret pieces of electro-acoustic music but since it uses a symbolic representations of the scores, it can bee seen as a system for more generally writing interaction with a temporal approach. Then, numerous other applications could be possible. This system can be used to create interactive multimedia scenarios (for theater shows or museum visits as examples) and also for adapting musical pieces or interactive multimedia documents for mutli-player applications or players with limited ability.

Solving the musical orchestration problem using multiobjective constrained optimization with a genetic local search approach

In this paper a computational approach of musical orchestration is presented. We consider orchestration as the search of relevant sound combinations within large instruments sample databases and propose two cooperating metaheuristics to solve this problem. Orchestration is seen here as a particular case of finding optimal constrained multisets on a large ensemble with respect to several objectives. We suggest a generic and easily extendible formalization of orchestration as a constrained multiobjective search towards a target timbre, in which several perceptual dimensions are jointly optimized. We introduce Orchidée, a time-efficient evolutionary orchestration algorithm that allows the discovery of optimal solutions and favors the exploration of non-intuitive sound mixtures. We also define a formal framework for global constraints specification and introduce the innovative CDCSolver repair metaheuristic, thanks to which the search is led towards regions fulfilling a set of musical-related requirements. Evaluation of our approach on a wide set of real orchestration problems is also provided.