Jônatas Manzolli

Roboser: A Real-World Composition System

We present a novel paradigm for the interactive composition and performance of music called Roboser consisting of a real-world device (i.e., a robot), its control software, and a composition engine that produces streams of MIDI data in real time. To analyze the properties of this framework, we present the application of Roboser to a learning mobile robot, called EmotoBot, that is controlled by the Distributed Adaptive Control (DAC) architecture. The EmotoBot composition is based on the generation of real-time sound events that express sensory, behavioral, and the internal states of the robot’s control model. We show that EmotoBot produces a complex set of sonic layers and quantify its ability to generate complex emergent sonic structures. We subsequently describe further applications of the Roboser framework to other interactive systems, including a large-scale interactive exhibition called Ada. Our results show the potential of the Roboser paradigm to define the central-processing stage of interactive composition systems. Moreover, Roboser provides a general framework for transforming information from real-world systems into complex sonic structures and as such constitutes a real-world composition system.

The eXperience Induction Machine: A New Paradigm for Mixed-Reality Interaction Design and Psychological Experimentation

The eXperience Induction Machine (XIM) is one of the most advanced mixed-reality spaces available today. XIM is an immersive space that consists of physical sensors and effectors and which is conceptualized as a general-purpose infrastructure for research in the field of psychology and human–artifact interaction. In this chapter, we set out the epistemological rational behind XIM by putting the installation in the context of psychological research. The design and implementation of XIM are based on principles and technologies of neuromorphic control. We give a detailed description of the hardware infrastructure and software architecture, including the logic of the overall behavioral control. To illustrate the approach toward psychological experimentation, we discuss a number of practical applications of XIM. These include the so-called, persistent virtual community, the application in the research of the relationship between human experience and multi-modal stimulation, and an investigation of a mixed-reality social interaction paradigm.

Vox Populi: Evolutionary Computation for Music Evolution

Publisher Summary This chapter introduces a new system, Vox Populi , based on evolutionary computation, for composing music in real time. A population of chords is properly codified according to the MIDI protocol and evolves by the application of genetic algorithms. A fitness criterion is defined to indicate the best chord in each generation, and this chord is selected as the next element in the sequence to be played. Each new generated chord is a new sound palette that a musician can use to continue the music evolution. Graphic controls provide user-friendly manipulation of the fitness and of the sound attributes. Evolutionary computation is used to stimulate the user with novel sounds, and it allows the user to respond. Associating the dynamic behavior of genetic algorithms with these tools for real-time interaction, Vox Populi becomes a musical instrument. But unlike a traditional instrument, Vox Populi is able to create its own sound raw material and to provide choice criteria simultaneously. All these features enhance the users music capabilities and mark this system as state of the art in computer music. Next, a general description of the main components of the computational environment and melodic, harmonic, and voice range criteria for musical fitness are defined. The chapter also explains the genetic encoding of notes and the evolutionary cycle for chord production.

An Investigation of Collective Human Behavior in Large-Scale Mixed Reality Spaces

Future mixed reality systems will need to support large numbers of simultaneous, nonexpert users at reasonable per-user costs if the systems are to be widely deployed within society in the short to medium term. We have constructed a prototype of such a system, an interactive entertainment space called Ada that was designed to behave like a simple organism. Using Ada we conducted two studies: the first assessing the effect of varying the operating parameters of the space on the collective behavior and attitudes of its users, and the second assessing the relationships among user demographics, behavior, and attitudes. Our results showed that small changes in the ambient settings of the environment have a significant effect on both user attitudes and behavior, and that the changes in user attitudes do not necessarily correspond to the environmental changes. We also found that individual user opinions are affected by demographics and reflected in overt behavior. Using these results, we propose some tentative guidelines for the design of future shared mixed reality spaces.

Ada: constructing a synthetic organism

Despite immense progress in neuroscience, we remain restricted in our ability to construct autonomous behaving robots that match the competence of even simple animals. The barriers to the realisation of this goal include: the lack of knowledge of system integration issues, engineering limitations and organisational constraints common to many research laboratories. In this paper we describe our approach to addressing these issues by constructing an artificial organism within the framework of the Ada project - a large-scale public exhibit for the Swiss Expo.02 national exhibition.

Technology and Digital Art: Creating video art with evolutionary algorithms

The boundaries of art are subjective, but the impetus for art is often associated with creativity, regarded with wonder and admiration along human history. Most interesting activities and their products are a result of creativity. The main goal of our approach is to explore new creative ways of editing and producing videos, using evolutionary algorithms. A creative evolutionary system makes use of evolutionary computation operators and properties and is designed to aid our own creative processes, and to generate results to problems that traditionally required creative people to solve. Our system is able to generate new videos or to help a user in doing so. New video sequences are combined and selected, based on their characteristics represented as video annotations, either by defining criteria or by interactively performing selections in the evolving population of video clips, in forms that can reflect editing styles. With evolving video, the clips can be explored through emergent narratives and aesthetics in ways that may reveal or inspire creativity in digital art.

Evolutionary computation applied to algorithmic composition

This paper presents an end-user interface that allows real time parametric control of sound events. It is an interactive environment in which evolutionary computation is applied to algorithmic composition. This system uses genetic algorithms to generate and evaluate a sequence of chords played as MIDI data. Melodic, harmonic and voice range fitness are used to control musical features. Based on the ordering of consonance of musical intervals, the notion of approximating a sequence of notes to its harmonically compatible note or tonal center is used. This method employs a fuzzy formalism and is posed as an optimization approach based on factors relevant to hearing music.

VR-RoBoser: real-time adaptive sonification of virtual environments based on avatar behavior

Until recently, the sonification of Virtual Environments had often been reduced to its simplest expression. Too often soundscapes and background music are predetermined, repetitive and somewhat predictable. Yet, there is room for more complex and interesting sonification schemes that can improve the sensation of presence in a Virtual Environment. In this paper we propose a system that automatically generates original background music in real-time called VR-RoBoser. As a test case we present the application of VR-RoBoser to a dynamic avatar that explores its environment. We show that the musical events are directly and continuously generated and influenced by the behavior of the avatar in three-dimensional virtual space, generating a context dependent sonification.

Application of an artificial immune system in a compositional timbre design technique

Computer generated sounds for music applications have many facets, of which timbre design is of groundbreaking significance. Timbre is a remarkable and rather complex phenomenon that has puzzled researchers for a long time. Actually, the nature of musical signals is not fully understood yet. In this paper, we present a sound synthesis method using an artificial immune network for data clustering, denoted aiNet. Sounds produced by the method are referred to as immunological sounds. Basically, antibody-sounds are generated to recognize a fixed and predefined set of antigen-sounds, thus producing timbral variants with the desired characteristics. The aiNet algorithm provides maintenance of diversity and an adaptive number of resultant antibody-sounds (memory cells), so that the intended aesthetical result is properly achieved by avoiding the formal definition of the timbral attributes. The initial set of antibody-sounds may be randomly generated vectors, sinusoidal waves with random frequency, or a set of loaded waveforms. To evaluate the obtained results we propose an affinity measure based on the average spectral distance from the memory cells to the antigen-sounds. With the validation of the affinity criterion, the experimental procedure is outlined, and the results are depicted and analyzed.

Artificial Abduction: A Cumulative Evolutionary Process

Can computers abduct, in the sense of proposing explanatory hypotheses to be further selected, as pointed out by Peirce (1940)? The present paper is certainly not devoted to providing a direct or suggestive answer to this question. However, a representative ensemble of computational mechanisms and concepts is shown to touch the essence of abduction, particularly when interactive evolutionary computation algorithms are considered. Probably the most famous and closest enquiry about the abductive capability of computers was formulated by Ada Lovelace, the friend and collaborator of Charles Babbage: can computers have anything to do with creativity? Lady Lovelace realized that Babbage’s ‘Analytical Engine’ — in essence, a design for a digital computer — could in principle ‘compose elaborate and scientific pieces of music of any degree of complexity or extent.’ But she insisted that the creativity involved in any elaborate pieces of music emanating from the Analytical Engine would have to be attributed not to the engine, but to the engineer (Boden 1998). She said: ‘The Analytical Engine has no pretensions whatever to originate anything. It can do [only] whatever we know how to order it to perform’ (Lovelace, quoted in Boden 1996: 84). That Analytical Engine has never been built, but Babbage recognized that, in principle, his machine was capable of playing games such as checkers and chess by looking forward to possible alternative outcomes, based on current potential moves. Since then, some pieces of art have been emerging from computers for many years. Artificial intelligence (AI) is the field within which much of this work takes place (Boden 1996; Bentley and Corne 2002). AI studies the nature of intelligence in general, and the methodology is devoted to enabling computers to do the sort of things that minds can do: seeing, speaking, storytelling, and logical or analogical thinking. The main goal is to understand, either for theoretical or practical purposes, how representational structures can generate behavior, and how intelligent behavior can emerge out of unintelligent behavior (Boden 1998).