Sytse Wierenga

Advanced interfaces to stem the data deluge in mixed reality: placing human (un)consciousness in the loop

We live in an era of data deluge and this requires novel tools to effectively extract, analyze and understand the massive amounts of data produced by the study of natural and artificial phenomena in many areas of research.

Social cooperation and competition in the mixed reality space eXperience Induction Machine XIM

Although the architecture of mixed reality spaces is becoming increasingly more complex, our understanding of human behavior in such spaces is still limited. Despite the sophisticated methods deployed in ethology and behavioral biology to track and analyze the actions and movements of animals, we rarely find studies that focus on the understanding of human behavior using such instruments. Here, we address this issue by analyzing social behavior and physical actions of multiple humans who are engaging in a game. As a paradigm of social interaction, we constructed a mixed reality football game in which two teams of two players have to cooperate and compete in order to win. This paradigm was deployed in the, so-called, eXperience Induction Machine (XIM), a human accessible, fully instrumented space that supports full body interaction in mixed reality without the need for body-mounted sensors. Our results show that winning and losing strategies can be discerned by specific behavioral patterns and proxemics. This demonstrates that mixed reality systems such as XIM provide new paradigms for the investigation of human social behavior.

Spatializing experience: a framework for the geolocalization, visualization and exploration of historical data using VR/AR technologies

In this study we present a novel ICT framework for the exploration and visualization of historical information using Augmented Reality (AR) and geolocalization. The framework facilitates the geolocalization of multimedia files, as well as their later retrieval and visualization through an AR paradigm in which a virtual reconstruction is matched to users positions and viewing angle. The main objective of the architecture is to enhance human-data interaction with cultural heritage content in outdoor settings and generate more engaging and profound learning experiences by exploiting information spatialization and sequencing strategies.

EFAA: a companion emerges from integrating a layered cognitive architecture

In this video, we present the human robot interaction generated by applying the DAC cognitive architecture [1] on the iCub robot. We demonstrate how the robot reacts and adapts to its environment within the context a continuous interactive scenario including different games. We emphasize as well that the artificial agent is maintaining a self-model in terms of emotions and drives and how those are expressed in order affect the social interaction. Categories and Subject Descriptors I.2.9 [Commercial robots and applications]: Layered Cognitive Architecture for Distributed and Adaptive Control of a social robot. General Terms Experimentation, Human Factors, Theory

re(PER)curso: an interactive mixed reality chronicle

re(PER)curso presents an interactive mixed reality narrative where two human performers – a percussionist and a dancer and a number of real-time synthetic actors including sonification, virtual cameras and an anthropomorphic avatar, explore the confluence of the physical and the virtual dimensions underlying existence and experience (Figure 1). The synthetic components of re(PER)curso are realized with computer generated graphics, automated moving light and stage control, video art, a synthetic music composition system called RoBoser [Manzolli and Verschure 2005], and an avatar embedded in a 3D graphic environment. The integration of all elements is realized through the multi-modal mixed reality system the eXperience Induction Machine (XIM) that is based on an earlier large scale public exhibition called Ada [Eng 2003]. XIM is controlled through a neuromorphic system that defines all the rules of interaction and performance dynamics and as a result the complete performance is synthesized in real-time and evolves without human intervention beyond that of the two human actors on the stage. re(PER)curso is an experiment in interactive narrative and explores the potential of virtual reality and augmented feedback technologies as tools for artistic expression. It expresses a general research strategy where the limits of advanced technologies are explored through their application in art. re(PER)curso is operated as an autonomous interactive installation that is augmented by 2 human performers. It is supported by a number of input devices that track and analyze the ongoing performance through cameras and microphones; controllers such as the synthetic composition engine RoBoser and output systems that include the large-scale real-time computer graphics, moving virtual and real cameras, and moving lights. Stage information obtained by the tracking systems is also projected onto the virtual world where it modulates the avatar’s behavior allowing it to adjust body position, posture and gaze to the physical world and to adjust properties of the virtual cameras.