Roberto Pugliese

Enactive cinema paves way for understanding complex real-time social interaction in neuroimaging experiments.

We outline general theoretical and practical implications of what we promote as enactive cinema for the neuroscientific study of online socio-emotional interaction. In a real-time functional magnetic resonance imaging (rt-fMRI) setting, participants are immersed in cinematic experiences that simulate social situations. While viewing, their physiological reactions—including brain responses—are tracked, representing implicit and unconscious experiences of the on-going social situations. These reactions, in turn, are analyzed in real-time and fed back to modify the cinematic sequences they are viewing while being scanned. Due to the engaging cinematic content, the proposed setting focuses on living-by in terms of shared psycho-physiological epiphenomena of experience rather than active coping in terms of goal-oriented motor actions. It constitutes a means to parametrically modify stimuli that depict social situations and their broader environmental contexts. As an alternative to studying the variation of brain responses as a function of a priori fixed stimuli, this method can be applied to survey the range of stimuli that evoke similar responses across participants at particular brain regions of interest.

Reality-based User Interface System (RUIS)

We introduce Reality-based User Interface System (RUIS), a platform that even novice programmers can easily use for prototyping novel user interfaces. We implemented a puzzle application with a 3D user interface employing modified Nintendo Wii Remotes as 6-DOF controllers to showcase some of the features that are possible with our platform: two-handed gestural interaction, natural interaction metaphors, and haptic feedback. We conducted a basic user study with 16 participants to evaluate the puzzle application and its 3D user interface. The results show how test subjects using the puzzle application tended to rate the 3D user interface to be more fun, intuitive, and realistic than a traditional mouse interface.

A framework for motion based bodily enaction with virtual characters

We propose a novel methodology for authoring interactive behaviors of virtual characters. Our approach is based on enaction, which means a continuous two-directional loop of bodily interaction. We have implemented the case of two characters, one human and one virtual, who are separated by a glass wall and can interact only through bodily motions. Animations for the virtual character are based on captured motion segments and descriptors for the style of motions that are automatically calculated from the motion data. We also present a rule authoring system that is used for generating behaviors for the virtual character. Preliminary results of an enaction experiment with an interview show that the participants could experience the different interaction rules as different behaviors or attitudes of the virtual character.

Game Experience When Controlling a Weak Avatar in Full-Body Enaction

In this paper we describe a motion-controlled game based on a paradigm of a player enacting the character, rather than a character mimicking the player’s action. Our hypothesis is that a controlling scheme based on the adaptation of a player to the way the avatar is able to perform actions, can result in a stronger presence and psychological bond to the character. The approach is based on previous studies showing that features, attitudes and behaviors of the digital representation of players in a virtual reality setting, alter the players self-perception in the virtual environment (Proteus effect). The interaction mechanism is inspired by enactive approach to cognition and embodied action. In a mini-game we explore effects of controlling a weak character on self-presence and identification with the avatar. We show that increasing degrees of effort in the controlled bodies resulted in different impressions of the physical state of the character. Additionally, we provide our interpretation of relation between game experience and the kinetic parameters and adaptation indicators extracted from the motion of the player and avatar. Finally, we address scenarios where this enaction-based approach to motion controlled avatar can find application.

Authoring rules for bodily interaction: from example clips to continuous motions

We explore motion capture as a means for generating expressive bodily interaction between humans and virtual characters. Recorded interactions between humans are used as examples from which rules are formed that control reactions of a virtual character to human actions. The author of the rules selects segments considered important and features that best describe the desired interaction. These features are motion descriptors that can be calculated in real-time such as quantity of motion or distance between the interacting characters. The rules are authored as mappings from observed descriptors of a human to the desired descriptors of the responding virtual character. Our method enables a straightforward process of authoring continuous and natural interaction. It can be used in games and interactive animations to produce dramatic and emotional effects. Our approach requires less example motions than previous machine learning methods and enables manual editing of the produced interaction rules.

Sonic trampoline: the effect of audio feedback on the user experience during an exercise of jumping

In this paper we examine the influence of auditory augmentation on the act of jumping on an elastic trampoline. To do so we have developed a system that interactively augments the sound of a trampoline when a user is jumping on it. The sound design is inspired by iconic jumping sounds from games and the synthesis engine allows for parametric control of the sound features. The sensing technology is based on a combination of motion tracking with depth camera and audio-based contact sensing between the feet and the trampoline. The system lets the user interactively control the sound by jumping. We conducted a study to evaluate the effect of manipulating the auditory feedback during the jumping exercises. Results show that our interactive sonification positively affects the user experience during the exercise and stimulates changes in the users behavior toward increased performance. Our system and study provide evidence that interactive sonification can act as a motivational tool in training and add an extra fun-factor to body-controlled games.

Sonic Trampoline: How Audio Feedback Impacts the User's Experience of Jumping

To investigate how auditory augmentation influences user jumping activity on an elastic trampoline, the authors developed a system to interactively augment the trampolines sound. The systems sound design is inspired by iconic jumping sounds from games, with sensing technology based on a combination of motion tracking with depth camera and audio-based contact sensing between the feet and trampoline. Results show that such interactive sonification stimulates changes in the users behavior toward increased performance during the exercise; it also positively affects the user experience.