Charles E. Hughes

Mixed reality in education, entertainment, and training

Transferring research from the laboratory to mainstream applications requires the convergence of people, knowledge, and conventions from divergent disciplines. Solutions involve more than combining functional requirements and creative novelty. To transform technical capabilities of emerging mixed reality (MR) technology into the mainstream involves the integration and evolution of unproven systems. For example, real-world applications require complex scenarios (a content issue) involving an efficient iterative pipeline (a production issue) and driving the design of a story engine (a technical issue) that provides an adaptive experience with an after-action review process (a business issue). This article describes how a multi-disciplinary research team transformed core MR technology and methods into diverse urban terrain applications. These applications are used for military training and situational awareness, as well as for community learning to significantly increase the entertainment, educational, and satisfaction levels of existing experiences in public venues.

High-dynamic-range still-image encoding in JPEG 2000

The raw size of a high-dynamic-range (HDR) image brings about problems in storage and transmission. Many bytes are wasted in data redundancy and perceptually unimportant information. To address this problem, researchers have proposed some preliminary algorithms to compress the data, like RGBE/XYZE, OpenEXR, LogLuv, and so on. HDR images can have a dynamic range of more than four orders of magnitude while conventional 8-bit images retain only two orders of magnitude of the dynamic range. This distinction between an HDR image and a conventional image leads to difficulties in using most existing image compressors. JPEG 2000 supports up to 16-bit integer data, so it can already provide image compression for most HDR images. In this article, we propose a JPEG 2000-based lossy image compression scheme for HDR images of all dynamic ranges. We show how to fit HDR encoding into a JPEG 2000 encoder to meet the HDR encoding requirement. To achieve the goal of minimum error in the logarithm domain, we map the logarithm of each pixel value into integer values and then send the results to a JPEG 2000 encoder. Our approach is basically a wavelet-based HDR still-image encoding method.

Networked virtual environments

The problem of resource bottlenecks is encountered in almost any distributed virtual environment or networked game. Whenever the demand for resources – such as network bandwidth, the graphics pipeline, or processing power – exceeds their availability, the resulting competition for the resources leads to a degradation of the system’s performance. In a typical client-server setup, for example, where the virtual world is managed by a server and replicated by connected clients which visualize the scene, the server must repeatedly transmit update messages to the clients. The computational power needed to select the messages to transmit to each client, or the network bandwidth limitations often allow only a subset of the update messages to be transmitted to the clients; this leads to a performance degradation and an accumulation of errors, e.g. a visual error based on the positional displacement of the moving objects. This thesis presents a scheduling algorithm, developed to manage the objects competing for system resources, that is able to achieve a graceful degradation of the system’s performance, while retaining an output sensitive behavior and being immune to starvation. This algorithm, called Priority Round-Robin (PRR) scheduling, enforces priorities based on a freely definable error metric, trying to minimize the overall error. The output sensitivity is a crucial requirement for the construction of scalable systems, and the freely definable error metric makes it suitable to be employed whenever objects compete for system resources, in client-server and peer-to-peer architectures as well. Therefore Priority Round-Robin scheduling is a substantial contribution to the development of distributed virtual environments and networked online-games.

How novelty search escapes the deceptive trap of learning to learn

A major goal for researchers in neuroevolution is to evolve artificial neural networks (ANNs) that can learn during their lifetime. Such networks can adapt to changes in their environment that evolution on its own cannot anticipate. However, a profound problem with evolving adaptive systems is that if the impact of learning on the fitness of the agent is only marginal, then evolution is likely to produce individuals that do not exhibit the desired adaptive behavior. Instead, because it is easier at first to improve fitness without evolving the ability to learn, they are likely to exploit domain-dependent static (i.e. non-adaptive) heuristics. This paper proposes a way to escape the deceptive trap of static policies based on the novelty search algorithm, which opens up a new avenue in the evolution of adaptive systems because it can exploit the behavioral difference between learning and non-learning individuals. The main idea in novelty search is to abandon objective-based fitness and instead simply search only for novel behavior, which avoids deception entirely and has shown prior promising results in other domains. This paper shows that novelty search significantly outperforms fitness-based search in a tunably deceptive T-Maze navigation domain because it fosters the emergence of adaptive behavior.

Applying mixed reality to entertainment

The lack of compelling content has relegated many promising entertainment technologies to laboratory curiosities. Although mixed-reality techniques show great potential, the entertainment business is not about technology. To penetrate these huge markets, MR technology must become transparent for the content to have full effect. To achieve this goal, we have devised a framework that lets us integrate concepts from disparate areas such as theme parks, theater, and film into a comprehensive research methodology. We believe that our framework, which has already helped us create content for MR entertainment systems, can provide these benefits to other developers as well.

Automatically generating visual syntax-directed editors

Since inexpensive computers possessing sophisticated graphics were introduced in the late 1970s, program development research has focused on syntax-directed editors that are based on the grammars of their underlying languages. The system presented here automatically generates object-oriented, syntax-directed editors for visual languages, which are described by a family of editing operations.

Evolving plastic neural networks with novelty search

Biological brains can adapt and learn from past experience. Yet neuroevolution, that is, automatically creating artificial neural networks (ANNs) through evolutionary algorithms, has sometimes focused on static ANNs that cannot change their weights during their lifetime. A profound problem with evolving adaptive systems is that learning to learn is highly deceptive. Because it is easier at first to improve fitness without evolving the ability to learn, evolution is likely to exploit domain-dependent static (i.e., nonadaptive) heuristics. This article analyzes this inherent deceptiveness in a variety of different dynamic, reward-based learning tasks, and proposes a way to escape the deceptive trap of static policies based on the novelty search algorithm. The main idea in novelty search is to abandon objective-based fitness and instead simply search only for novel behavior, which avoids deception entirely. A series of experiments and an in-depth analysis show how behaviors that could potentially serve as a stepping stone to finding adaptive solutions are discovered by novelty search yet are missed by fitness-based search. The conclusion is that novelty search has the potential to foster the emergence of adaptive behavior in reward-based learning tasks, thereby opening a new direction for research in evolving plastic ANNs.

The Potential of Simulated Environments in Teacher Education: Current and Future Possibilities

The future of virtual environments is evident in many fields but is just emerging in the field of teacher education. In this article, the authors provide a summary of the evolution of simulation in the field of teacher education and three factors that need to be considered as these environments further develop. The authors provide a specific example of the work at two universities that use a specific virtual environment, TLE TeachLivE™, in teacher education. This environment is already being used in teacher preparation at 32 universities to collaboratively find ways to enhance teacher practice while using a standardized tool often found in medicine, business and military training, and virtual simulation.

Shared virtual worlds for education: the ExploreNet experiment

Abstract.ExploreNet is an experimental environment for creating and delivering networked “virtual worlds.” This systems style of user interaction was inspired by the concept of a “habitat” as first articulated in the LucasFilms Habitat system. Players enter and interact in a habitat via their animated alter egos, called “avatars.” Habitats may be created for many purposes, including social interaction, entertainment and education. Our focus has been to facilitate the creation of habitats in which virtual communities of learners and mentors interact. This paper presents details of the current ExploreNet system, including its user interface, the means it provides for creating complex behaviors, details of its implementation, the outcomes of several experiments using this system, and our plans for its natural migration to a World Wide Web-based system.

Providing Real-time Feedback for Student Teachers in a Virtual Rehearsal Environment

Research in learning analytics and educational data mining has recently become prominent in the fields of computer science and education. Most scholars in the field emphasize student learning and student data analytics; however, it is also important to focus on teaching analytics and teacher preparation because of their key roles in student learning, especially in K-12 learning environments. Nonverbal communication strategies play an important role in successful interpersonal communication of teachers with their students. In order to assist novice or practicing teachers with exhibiting open and affirmative nonverbal cues in their classrooms, we have designed a multimodal teaching platform with provisions for online feedback. We used an interactive teaching rehearsal software, TeachLivE, as our basic research environment. TeachLivE employs a digital puppetry paradigm as its core technology. Individuals walk into this virtual environment and interact with virtual students displayed on a large screen. They can practice classroom management, pedagogy and content delivery skills with a teaching plan in the TeachLivE environment. We have designed an experiment to evaluate the impact of an online nonverbal feedback application. In this experiment, different types of multimodal data have been collected during two experimental settings. These data include talk-time and nonverbal behaviors of the virtual students, captured in log files; talk time and full body tracking data of the participant; and video recording of the virtual classroom with the participant. 34 student teachers participated in this 30-minute experiment. In each of the settings, the participants were provided with teaching plans from which they taught. All the participants took part in both of the experimental settings. In order to have a balanced experiment design, half of the participants received nonverbal online feedback in their first session and the other half received this feedback in the second session. A visual indication was used for feedback each time the participant exhibited a closed, defensive posture. Based on recorded full-body tracking data, we observed that only those who received feedback in their first session demonstrated a significant number of open postures in the session containing no feedback. However, the post-questionnaire information indicated that all participants were more mindful of their body postures while teaching after they had participated in the study.