G. De Poli

Communicating expressiveness and affect in multimodal interactive systems

Multisensory integrated expressive environments is a framework for mixed reality applications in the performing arts such as interactive dance, music, or video installations. MIEEs address the expressive aspects of nonverbal human communication. We present the multilayer conceptual framework of MIEEs, algorithms for expressive content analysis and processing, and MIEEs-based art applications.

Modeling and control of expressiveness in music performance

Expression is an important aspect of music performance. It is the added value of a performance and is part of the reason that music is interesting to listen to and sounds alive. Understanding and modeling expressive content communication is important for many engineering applications in information technology. For example, in multimedia products, textual information is enriched by means of graphical and audio objects. In this paper, we present an original approach to modify the expressive content of a performance in a gradual way, both at the symbolic and signal levels. To this purpose, we discuss a model that applies a smooth morphing among performances with different expressive content, adapting the audio expressive character to the users desires. Morphing can be realized with a wide range of graduality (from abrupt to very smooth), allowing adaptation of the system to different situations. The sound rendering is obtained by interfacing the expressiveness model with a dedicated postprocessing environment, which allows for the transformation of the event cues. The processing is based on the organized control of basic audio effects. Among the basic effects used, an original method for the spectral processing of audio is introduced.

The MEGA Project: Analysis and Synthesis of Multisensory Expressive Gesture in Performing Art Applications

This article presents a survey of the research work carried out within the framework of the European Union-IST project MEGA (Multisensory Expressive Gesture Applications, November 2000 – October 2003; www.megaproject.org). First, the article introduces a layered conceptual framework for analysis and synthesis of expressive gesture. Such a framework represents the main methodological foundation upon which the MEGA project built its own research. A brief overview of the achievements of research in expressive gesture analysis and synthesis is then provided: these are the outcomes of some experiments that were carried out in order to investigate specific aspects of expressive gestural communication. The work resulted in the design and development of a collection of software libraries integrated in the MEGA System Environment (MEGASE) based on the EyesWeb open platform (www.eyesweb.org).

Elimination of delay-free loops in discrete-time models of nonlinear acoustic systems

Nonlinear acoustic systems are often described by means of nonlinear maps which act as instantaneous constraints on the solutions of a system of linear differential equations. This description leads to discrete-time models exhibiting non-computable loops. This paper presents a solution to this computability problem by means of geometrical transformation of the nonlinearities and algebraic transformation of the time-dependent equations. The proposed leads to stable and accurate simulations even at relatively low sampling rates.

Sound authoring tools for future multimedia systems

A framework for authoring non-speech sound objects in the context of multimedia systems is proposed. The goal is to design specific sounds and their dynamic behavior in such a way that they convey dynamic and multidimensional information. Sounds are designed using a three-layer abstraction model: physically-based description of sound identity, signal-based description of sound quality, perception- and geometry-based description of sound projection in space. The model is validated with the aid of an experimental tool where manipulation of sound objects can be performed in three ways: handling a set of parameter control sliders, editing the evolution in time of compound parameter settings, via client applications sending their requests to the sounding engine.

Adding expressiveness to musical messages

A system to add expressiveness to musical messages has been developed, starting from the results of acoustic and perceptual analyses. The system allows to obtain different performances, by modifying the acoustic parameters of a given neutral performance. The modification of the input performance is performed by a model that uses the hierarchical segmentation of the musical organization. For every hierarchical level, opportune curves are applied to the principal acoustic parameters. Levels self-similarity is the main criteria to construct the curves. The modular structure of the system defines an open architecture, where the rendering steps can be realized both with synthesis and post-processing techniques. Different synthesis techniques, like FM, physical models or wavetable have been explored.