Myriam Desainte-Catherine

Interactions and systems for augmenting a live dance performance

The context of this work is to develop, adapt and integrate augmented reality related tools to enhance the emotion involved in cultural performances. Part of the work was dedicated to augmenting a stage in a live performance, with dance as an application case. In this paper, we present a milestone of this work, an augmented dance show that brings together several tools and technologies that were developed over the projects lifetime. This is the result of mixing an artistic process with scientific research and development. This augmented show brings to stage issues from the research fields of Human-Machine Interaction (HMI) and Augmented Reality (AR). Virtual elements are added on stage (visual and audio) and the dancer is able to interact with them in real-time, using different interaction techniques. The originality of this work is threefold. Firstly, we propose a set of movement-based interaction techniques that can be used independently on stage or in another context. In this set, some techniques are direct, while others go through a high level of abstraction. Namely, we performed movement-based emotion recognition on the dancer, and used the recognized emotions to generate emotional music pieces and emotional poses for a humanoid robot. Secondly, those interaction techniques rely on various interconnected systems that can be reassembled. We hence propose an integrated, interactive system for augmenting a live performance, a context where system failure is not tolerated. The final system can be adapted following the artists preferences. Finally, those systems were validated through an on field experiment - the show itself - after which we gathered and analyzed the feedback from both the audience and the choreographer.

A Framework for Composition, Verification and Real-Time Performance of Multimedia Interactive Scenarios

Interactive Scores (IS) is a formalism for composing and performing interactive multimedia scenarios. In IS, the composer defines temporal relations (TRs) between temporal objects (TOs) in order to specify the temporal organization of the scenario. During execution, the performer may trigger interaction points to modify the star/stop times of TOs, while the system guarantees that all the TRs are satisfied. IS is implemented in the tool I-SCORE and its semantics is formally defined as a Hierarchical Time Stream Petri Net (HTSPN). However, this model is not able to represent branching behaviors that are necessary to properly deal with applications such as video games and museum installations. Moreover, HTSPN does not provide tools for the automatic verification of critical properties of scenarios. In this work we define a semantics for IS based on Timed Automata (TA) and we show that such model yields to a complete framework to compose, verify and execute interactive scenarios. More precisely, we show that: 1) our model is able to deal with conditional statements in IS; 2) efficient verification techniques can be now used to reason about the written scenarios; and 3) our model allows for a directly implementation on a reconfigurable device, thus guaranteeing a real-time performance.

Modelling Data Processing for Interactive Scores Using Coloured Petri Nets

I-score is a system for the composition and execution of interactive multimedia scores. It uses Hierarchical Time Stream Petri Nets (HTSPN) to build an execution model of the scores. Nowadays, composers have increasingly needed to represent and manipulate complex data in their multimedia scenarios. However, HTSPN formalism does not allow to handle data. In this work, we propose a model to execute interactive multimedia scores based on Coloured Petri Nets (CPN). Our work extends the current execution model of i-score with the capability to handle complex data. Our approach consists in developing CPN modules for reading, appending and reversing audio files. We use CPN Tools for prototyping, simulating and verifying our model and discuss how to represent fundamental signal processing functions, conditionals or loops.

Analysis method to approximate the spectral density of noises

Usual noise models represent sounds with filtered white noise, and extract only the spectral envelope from the sounds analyzed. Psychoacoustic experiments have shown the ability of humans to discriminate noise bands with different spectral densities. This property may be taken into account for the mathematical representation of noises. We propose an original method to approximate this new parameter. The limitations of the discrete short-term transform lead us to consider a new approach. This technique is based on statistical studies of the intensity fluctuations which are theoretically related to the spectral density. An algorithm is detailed and its parameters are discussed. Experiments show the quality and the limitations of this method. Although it may be improved, this estimation technique is accurate enough to be applied in a complete analysis/synthesis model for noisy sounds.