Darin E. Hughes

Mixed reality in education, entertainment, and training

Transferring research from the laboratory to mainstream applications requires the convergence of people, knowledge, and conventions from divergent disciplines. Solutions involve more than combining functional requirements and creative novelty. To transform technical capabilities of emerging mixed reality (MR) technology into the mainstream involves the integration and evolution of unproven systems. For example, real-world applications require complex scenarios (a content issue) involving an efficient iterative pipeline (a production issue) and driving the design of a story engine (a technical issue) that provides an adaptive experience with an after-action review process (a business issue). This article describes how a multi-disciplinary research team transformed core MR technology and methods into diverse urban terrain applications. These applications are used for military training and situational awareness, as well as for community learning to significantly increase the entertainment, educational, and satisfaction levels of existing experiences in public venues.

Spatial Perception and Expectation: Factors in Acoustical Awareness for Mout Training

Abstract : Mixed Reality (MR), and its predecessor Virtual Reality (VR), has been primarily viewed as a visual science, with much less attention given to the other senses, despite clear evidence of their importance, especially audio. In fact, in military operation in urban environments, audio is often a more primary sense than vision, providing a soldier with an early warning system that needs to be honed and trained. Our research program, in contrast, treats the auditory sense as an equal to the visual. The consequences are MR experiences that have much greater impact than those in which audio is just an after thought. However, given the depth and breadth of graphics research, we are compelled to learn from this mature area. Thus, we are constantly striving to find results from graphics research that have useful analogies in the audio domain. Lessons learned from these analogies, especially as concern peoples perception expectations, are the focus of this paper.

Virtual Learning Environments for Students with Disabilities: A Review and Analysis of the Empirical Literature and Two Case Studies.

Students with autism spectrum disorder (ASD) show varying levels of impairment in social skills situations. Interventions have been developed utilizing virtual environments (VEs) to teach and improve social skills. This article presents a systematic literature review of peer-reviewed journal articles focusing on social interventions in VEs involving K-12th grade students with ASD. This exhaustive analysis across four major online databases was guided by operational terms related to intervention type and K-12 students with ASD. The empirical search yielded a very narrow body of literature (n=19) on the use of VEs as social skill interventions for students with ASD. Two case study examples of experiments exploring the use of VEs and students with ASD are presented to illustrate possible applications of this technology.

Integrating and Delivering Sound Using Motion Capture and Multi-tiered Speaker Placement

Creating effective and compelling soundscapes for simulations is a challenging process that requires non-traditional tools and techniques outside the scope of standard production methods. In an immersive simulation, sound is at least as important as graphics; auditory cues can be heard from behind walls, around corners, and out of the line of sight. This paper describes a novel approach to interactive 3D sound design utilizing vision-based motion capture and multi-tiered, configurable loudspeaker delivery.

Improving perspective taking and empathy in children with autism spectrum disorder

This paper discusses the design, implementation, and evaluation of a serious game intended to reinforce applied behavior analysis (ABA) techniques used with children with autism spectrum disorder (ASD) by providing a low cost and easily accessible supplement to traditional methods. The goal is develop a safe environment for social exploration and learning that boosts the childs confidence while providing calming mechanisms. Games increase childrens motivation and thus increase the rate of learning in computer mediated environments. Furthermore, children with ASD are able to understand basic emotions and facial expressions in avatars more easily than in real-world interactions.

Enhancing protective role-playing behaviors through avatar-based scenarios

This paper describes a virtual environment, called CollegeLiVE that presents users with an experience centered on realistic social interactions that first-year college students likely encounter. The virtual characters and settings present participants with typical situations that can challenge them to address peer pressure surrounding alcohol consumption and related situations from walking or driving while inebriated as well as determining how best to handle undue pressure placed on others to drink or engage in sexual activity. The employment of digital puppeteering in addition to agent-based control of virtual characters (avatars) makes this virtual environment potentially helpful in enhancing protective skills meant to reduce risky drinking practices and associated behaviors. The presence of a human-in-the-loop avoids the shortcomings of pure agent-based control, affording an experience in which avatars respond appropriately to both verbal and nonverbal cues in dialogues with a user. Reflection (self assessment of how one handled different situations) and after-action review (feedback about ones performance from a coach) enhances learning from the experience.

MeEmo - Using an Avatar to Improve Social Skills in Children with ASD.

For individuals affected by Autism Spectrum Disorder (ASD), the inability to make eye contact is a significant barrier to their engagement in social environments. This lack of eye contact limits their ability to read social and emotional cues exhibited through facial expressions resulting in a corresponding decrease in social engagement. The use of interactive virtual environments (VEs) as a therapeutic protocol is a growing field of study. In recent studies, individuals with ASD were placed in VEs and engaged with avatars controlled by a human in the background, resulting in improvements in eye contact and engagement for some subjects. This paper is the first in a series of experiments exploring the potential of virtual avatars controlled through software agency, rather than human control, as a therapeutic tool for ASD. This paper examines if a subject could learn to make eye contact with an avatar and consequently recognize and respond to emotional cues expressed by the avatar. Results indicate that children with ASD can learn to recognize the emotional cues of the virtual avatar, and that their reactions to the avatar’s needs as well as their eye contact with the avatar improved over the course of the VE experiment. This study sets the stage for future exploration into therapeutic use of agent-based virtual avatars, including transference of emotional cues from avatars to humans in the real world.

Mixed Reality Space Travel for Physics Learning

In this paper we describe research being conducted on a mixed reality simulation called MEteor that is designed for informal physics learning in science centers. MEteor is a 30 x 10 foot floor area where participants use their bodies to interact with projected astronomical imagery. Participants walk and run across the floor to simulate how objects move in space, and to enact basic physics principles. Key to the success of this learning environment is an interface scheme that supports the central metaphor of “child as asteroid.” Using video data collected in our studies we examine the extent to which feedback mechanisms and interface conventions strengthened the metaphorical connection, and we describe ways the interaction design can be improved for future iterations.

ChronoLeap: The Great World’s Fair Adventure

ChronoLeap: The Great World’s Fair Adventure utilizes the educational potential of immersive 3D virtual venues for children and early adolescents between 9 and 13. Virtual reality environments transport the mind beyond the 2D bounds of text or photographs; they engage the imagination and can be a powerful tool for conveying educational content [1]. ChronoLeap leverages these innate qualities and weaves together the individual threads of single disciplines into a multi-disciplinary tapestry of web-based exploration through the 1964/65 New York World’s Fair. Through their myriad of pavilions and exhibits, World Fairs offer links to science, technology, engineering, mathematics, art and humanities topics. ChronoLeap provides an immersive 3D environment with highly accurate and detailed models, and merges it with games and themes designed to provide users an educational STEAM environment. The project is a collaborative effort between the University of Central Florida, Queens Museum of Art and New York Hall of Science.

Interconnections Revisiting the Future

Virtual reality (VR) transports the mind beyond the two-dimensional bounds of text and photographs; it engages the imagination and forms visual and cognitive links. VR can free participants from stereotyped bounds projected by society. Interconnections: Revisiting the Future applies these innate qualities of virtual worlds to weave together individual threads of singular disciplines into a multidisciplinary tapestry of exploration. The authors are creating an accurately modeled 1964–1965 New York World’s Fair where users can freely explore 140+ pavilions set on 660 virtual square acres. The myriad of pavilions offer links to multiple disciplines—science, engineering, technology, national and international political/cultural affairs, art, history, and architecture. The project provides on-site museum experiences with its partners, the New York Hall of Science and Queens Museum of Art. The three-dimensional virtual Fair environment serves as a central portal that links together not only subjects within that environment but also experiences at these partnering institutions.Virtual reality (VR) transports the mind beyond the two-dimensional bounds of text and photographs; it engages the imagination and forms visual and cognitive links. VR can free participants from st...