Man Lok Yau

Embodied mobile agents

The move in many societies toward individuals having multiple networked computational devices -- workstations, notebooks computers, mobile phones, PDAs - radically changes the ways in which people engage those devices. However, we lack interaction paradigms that enable a coherent experience across these technologies. One possible approach to this problem involves the use of embodied mobile agents (EMAs), that is, graphically animated, autonomous or semi-autonomous software systems that can migrate seamlessly from one computational device to another. This paper describes an interactive museum exhibit that was implemented with EMAs, discusses the opportunities and challenges presented by this new form of agent, and consider other potential applications for EMAs. While not a universal solution to challenges of interacting with heterogeneous networks of devices, embodied mobile agents can help to provide a coherent user experience across multiple computational devices.

Dreaming of adaptive interface agents

This interactive project uses the metaphor of human sleep and dreaming to present a novel paradigm that helps address problems in adaptive user interface design. Two significant problems in adaptive interfaces are: interfaces that adapt when a user does not want them to do so, and interfaces where it is hard to understand how it changed during the process of adaptation. In the project described here, the system only adapts when the user allows it to go to sleep long enough to have a dream. In addition, the dream itself is a visualization of the transformation of the interface, so that a person may see what changes have occurred. This project presents an interim stage of this system, in which an autonomous agent collects knowledge about its environment, falls asleep, has dreams, and reconfigures its internal representation of the world while it dreams. People may alter the agents environment, may prevent it from sleeping by making noise into a microphone, and may observe the dream process that ensues when it is allowed to fall asleep. By drawing on the universal human experience of sleep and dreaming, this project seeks to make adaptive interfaces more effective and comprehensible.

The EcoRaft project: a multi-device interactive graphical exhibit for learning about restoration ecology

The EcoRaft Project, an interactive installation designed to help children learn about restoration ecology, allows participants to engage physically with animated agents via a natural and intuitive interface. This physical engagement occurs when the agents transfer seamlessly from stationary computers to mobile devices, on which the agents are realized as quasi-physical manifestations. Utilizing tablet PCs to act simultaneously as objects in the physical world and as mobile virtual spaces, the system incorporates embodied mobile agents that increase levels of engagement. The project has been publicly shown at several venues, where over 2000 participants interacted with the system. This paper presents initial evaluation results based on interviews with participants indicating that the embodied, physical interaction in this installation leads to participant engagement and collaboration, and enhanced educational effectiveness.

Richly connected systems and multi-device worlds

Many human activities now take place in settings that include several computational devicessuch as desktop computers, laptops, and mobile phonesin the same physical space. However, we lack interaction paradigms that support a coherent experience across these collocated technologies and enable them to work effectively as systems. This article presents a conceptual framework for building richly connected systems of collocated devices, and offers two implemented examples of interactive virtual worlds built on this framework. Aspects of this framework include multiple channels of real and apparent connectivity among devices: for example, multiple kinds of data networking, cross-device graphics and sound, and embodied mobile agents that inhabit the multi-device system. In addition, integration of the system with the physical world helps bridge the gap between devices. We evaluate the framework in terms of the types of user experiences afforded and enabled by the implemented systems. We also present a number of lessons learned from this evaluation regarding how to develop richly connected systems using heterogeneous devices, as well as the expectations that users bring to this kind of system. The core contribution of this paper is a novel framework for collocated multi-device systems; by presenting this framework, this paper lays the groundwork for a wide range of potential applications.

A Participatory Simulation for Informal Education in Restoration Ecology

Constructivist pedagogical approaches have become common in many science curricula. However, while sciences such as physics and chemistry lend themselves to compelling opportunities for interaction (explosions, reactions, objects in motion), certain systems sciences are more challenging for learners to engage with on a short time scale. Applying constructivist, discovery-based methods to the teaching of restoration ecology presents a number of difficulties, including the large amount of time and space over which ecological processes occur, limitations on experiments that can be performed in classroom contexts, and limited access to certain ecological settings. To address these difficulties, a team of computer scientists, ecologists, and educators created the EcoRaft project, a participatory simulation that allows children to learn about restoration ecology by playing the role of ecologists collaborating to restore a virtual rain forest ecosystem. Through qualitative evaluations of the EcoRaft project in multiple contexts, this article explores (1) how to design simulated environments more effectively as a means of promoting collaborative, discovery-based learning of scientific concepts, and (2) how to do so in ways that balance the need for integrating content area knowledge with interactive paradigms that engage young participants.

Heterogeneous character animation: how to make an interactive character jump between stationary and mobile graphical computing platforms

This sketch presents a method for enabling a real-time animated character to appear to jump between a stationary computer screen and a mobile graphical device. This heterogeneous character animation character animation that takes place across two or more networked and collocated graphical devices is a key element in the Virtual Raft Project, which is being shown in the Emerging Technologies program at SIGGRAPH 2005. The crossplatform jump occurs when a participant brings a Tablet PC (a “virtual raft”) up to a stationary computer (a “virtual island”). The character’s jump appears very simple to the participant, but technically it involves precise coordination of numerous technical and design elements autonomous behavior, interactive animation, real-world sensing, inter-device communication, interface design, physical set construction and graphical and sound effects. Approximately 250 participants have interacted with the Virtual Raft Project in groups of ten to twenty at a time; a video of these interactions accompanies this sketch and demonstrates how the characters jump from virtual island to virtual raft. Effective heterogeneous character animation could make possible novel forms of entertainment, education and new media art.

Persistence and propagation of shadow direction in mobile and multi-device graphics

Multi-Device Virtual Environments (MDVEs) - in which several collocated graphical computing devices integrate together to form a single virtual world - are a novel area for applications in entertainment, education and simulation. MDVEs composed of stationary and mobile devices, though, pose new challenges in the area of lighting design. Traditional computational lighting design techniques are static and do not take into account physical movement of the device. However, static light design does not work well in mobile devices; for example, the shadows in the environment remain static even if the user rotates the device. This paper addresses this challenge by attaching sensing devices to mobile computers to detect their physical movements. The MDVE then uses this feedback to adjust the virtual light source so that it remains consistent with how light sources work in real world. This lighting model may then be propagated via IrDA among the devices in the MDVE. By improving lighting in MDVEs, this research seeks to bridge the gap between the real world and a virtual world and to help enhance the believability of MDVEs.

Interactive lighting design for multi-device virtual environments

It is increasingly common for multiple computing devices to operate in close proximity. As these devices begin to work together more and more, it may be useful to see a collection of devices as a multi-device system rather than as a group of devices that function separately. One form of multi-device system is the multi-device virtual environment (MDVE), in which several devices work together to create the illusion of a graphical world that stretches seamlessly across their screens. MDVEs can potentially be useful for entertainment, education and industrial applications. There are a number of challenges involved in creating believable and engaging MDVEs and animated agents that exist on them, including areas such as graphics, sound, interaction paradigms and autonomous behavior. This poster presents a technique for improving multi-device virtual environment by enabling the different devices to share the same light source properties, thereby unifying the visual representations on each device.