Christopher B. Stapleton

Mixed reality in education, entertainment, and training

Transferring research from the laboratory to mainstream applications requires the convergence of people, knowledge, and conventions from divergent disciplines. Solutions involve more than combining functional requirements and creative novelty. To transform technical capabilities of emerging mixed reality (MR) technology into the mainstream involves the integration and evolution of unproven systems. For example, real-world applications require complex scenarios (a content issue) involving an efficient iterative pipeline (a production issue) and driving the design of a story engine (a technical issue) that provides an adaptive experience with an after-action review process (a business issue). This article describes how a multi-disciplinary research team transformed core MR technology and methods into diverse urban terrain applications. These applications are used for military training and situational awareness, as well as for community learning to significantly increase the entertainment, educational, and satisfaction levels of existing experiences in public venues.

Applying mixed reality to entertainment

The lack of compelling content has relegated many promising entertainment technologies to laboratory curiosities. Although mixed-reality techniques show great potential, the entertainment business is not about technology. To penetrate these huge markets, MR technology must become transparent for the content to have full effect. To achieve this goal, we have devised a framework that lets us integrate concepts from disparate areas such as theme parks, theater, and film into a comprehensive research methodology. We believe that our framework, which has already helped us create content for MR entertainment systems, can provide these benefits to other developers as well.

Human Experience Modeler: Context-Driven Cognitive Retraining to Facilitate Transfer of Learning

We describe a cognitive rehabilitation mixed-reality system that allows therapists to explore natural cuing, contextualization, and theoretical aspects of cognitive retraining, including transfer of training. The Human Experience Modeler (HEM) mixed-reality environment allows for a contextualized learning experience with the advantages of controlled stimuli, experience capture and feedback that would not be feasible in a traditional rehabilitation setting. A pilot study for testing the integrated components of the HEM is discussed where the participant presents with working memory impairments due to an aneurysm.

Applying Pedagogy during Game Development to Enhance Game-Based Learning

“Serious games” are emerging as an important outgrowth of the video gaming industry. Entertaining games, such as Flight Simulator and SimCity, are already in use in schools and universities across the country, and the number of serious games designed specifically for training and education is also on the rise. Advances in video game production, however, are far outpacing research on its design and effectiveness. Relatively little is still known about methods for optimizing the game design process or game-based learning.

Interactive imagination: Tapping the emotions through interactive story for compelling simulations

Most compassionate parents, teachers, or coaches will echo the belief that you must spark the imagination or touch the heart to teach the mind, train the body, or inspire a sense of wonder. This is also the storytellers craft. How can we use story within interactive simulations to better teach, train, or inspire? Now that science and technology can make simulations more realistic, how can art make them more compelling through interactive fiction? The key is to use story to tap the depths of emotions, engaging the users desire for exploration, learning, challenge, and adventure. In the new domain of training, story becomes the means more than the end. Can the compelling art of story transition from the passive media of motion pictures to the nonlinear interactivity of simulation? This is venturing beyond the reactive branching of cause-and-effect games or choose-your-own-ending adventure stories. This process is about the unpredictable expressiveness of audiences exchanging discourse with the author mediated through the digital media - the story engine.

Believing is seeing: cultivating radical media innovations

A major challenge to making creative leaps in media innovations is demonstrating the magnitude of human impact that each new media technology will bring. The emerging media of mixed reality - the blending of real and synthetic environments and objects - has melted the boundaries between physical and virtual realities, letting us interweave simulated characters and scenes into real-world experiences. So why do we treat mixed reality as we do more limiting media technology such as virtual reality, television, cinema, radio, or print media? A new vision might borrow from tradition, but by making creative leaps, we can find the magical power that lies within.

MIXED FANTASY: exhibition of entertainment research for mixed reality

This exhibition presents a Mixed Reality experience that applies basic research to the media industries of entertainment, training and informal education. Contributing areas of research include tracking, registration, rendering, real-time and distributed algorithms, and adaptations of existing as well as the creation of new forms of artistic convention. 1. Introduction: Why Entertainment? The power of entertainment stretches far beyond venues for amusement. Entertainment is a language unto itself affecting world culture, world economy and even world opinion. Entertainment is able to excite, motivate, satiate, communicate and inspire. This power is being applied to all aspects of life including learning, training, designing, communicating and collaborating for purposes ranging from social engagements to high powered teams making world decisions. The power of Mixed Reality provides us with the ability to combine the fantastical worlds of the imagination through virtual reality and seamlessly merge these with the compelling aspects of reality. MR envelops the most important aspect of reality-other people. The participants bring the most important part of Mixed Reality—the imagination. The Media Convergence Laboratory’s exhibition of

Melting the Boundaries Between Fantasy and Reality

Technology goes beyond visual effects to produce alternative realities for immersive entertainment experiences. The art and science of immersive entertainment illusions engage all the senses in every direction and dimension. The theme park is an extreme example of immersive entertainment, wherein experiential entertainment designers and scenario authors project the audience into an experienced reality, whether actual or imagined. It is unparalleled by other forms of entertainment because it can sustain this illusion for thousands of people for a few minutes, a full day, or even a week. With that amount of scrutiny, theme parks present a challenge in integrating computer-generated content for experiential entertainment. Seamlessly blending the physical reality and the authors virtual fantasy is critical in creating a successful audience experience. This illusion is only effective when the author can engage and direct the audiences imaginary reality.

Imagination: The third reality to the virtuality continuum

Both the art and science of the imagination have integral roles in defining compelling Mixed Reality (MR) experiences. In this paper we posit that the audience members own imagination is an essential third kind of input in defining the full virtuality continuum for MR. It is traditionally accepted that there are two experiential inputs in MR incorporating a combination of stimuli of the real world as well as from virtual artifacts (typically from computers). Using a case study of a MemoryScape Prototype for the Maitland Holocaust Museum, we explore how, in addition to reality and augmented/virtual reality, imagination artistically and scientifically serves as an important third reality to the virtuality continuum to achieve the experience designers intent for the audiences perception of MR experiences.

Transforming Lives Through Story Immersion: Innovation of Aphasia Rehabilitation Therapy through Storytelling Learning Landscapes

Aphasia is a disease that renders its victims unable to effectively use language. Evidence supports the efficacy of treatment for aphasia yet the effectiveness or transferability of learned communicative abilities to everyday conversation continues to be investigated. In this paper we explore an alternative approach to aphasia treatment based on the art and science of storytelling. Inherent in storytelling are the motivations to share an experience, the cognitive abilities to organize story, and the language system to convey the experience. This approach is based on decades of research in aphasia therapy and immersive storytelling (in other fields) and has been used to engage a subjects creativity and emotions to produce transformative results in real life. We report on early, promising results that could radically innovate the rehabilitative practice of aphasia.