Brent Cowan

Spatial sound for video games and virtual environments utilizing real-time GPU-based convolution

The generation of spatial audio is computationally very demanding and therefore, accurate spatial audio is typically overlooked in games and virtual environments applications thus leading to a decrease in both performance and the users sense of presence or immersion. Driven by the gaming industry and the great emphasis placed on the visual sense, consumer computer graphics hardware (and the graphics processing unit in particular), has greatly advanced in recent years, even outperforming the computational capacity of CPUs. This has allowed for real-time, interactive realistic graphics-based applications on typical consumer-level PCs. Despite the many similarities between the fields of spatial audio and computer graphics, computer graphics and image synthesis in particular, has advanced far beyond spatial audio given the emphasis placed on the generation of believable visual cues over other perceptual cues including auditory. Given the widespread use and availability of computer graphics hardware as well as the similarities that exist between the fields of spatial audio and image synthesis, this work investigates the application of graphics processing units for the computationally efficient generation of spatial audio for dynamic and interactive games and virtual environments. Here we present a real-time GPU-based convolution method and illustrate its superior efficiency to conventional, software-based, time-domain convolution.

A Survey of Frameworks and Game Engines for Serious Game Development

Given the sparsity of standard game engines and frameworks for serious game development, developers of serious games typically rely on entertainment-based game development tools. However, given the large number of game engines and frameworks dedicated to entertainment game development, deciding on which tool to employ may be difficult. A literature review that examined the frameworks and game engines used to develop serious games was recently conducted. Here, a list of the most commonly identified frameworks and game engines and a summary of their features is provided. The results presented provide insight to those seeking tools to develop serious games.

GPU-based real-time acoustical occlusion modeling

In typical environments, the direct path between a sound source and a listener is often occluded. However, due to the phenomenon of diffraction, sound still reaches the listener by “bending” around an obstacle that lies directly in the line of straight propagation. Modeling occlusion/diffraction effects is a difficult and computationally intensive task and thus generally ignored in virtual reality and videogame applications. Driven by the gaming industry, consumer computer graphics hardware and the graphics processing unit (GPU) in particular, have greatly advanced in recent years, outperforming the computational capacity of central processing units. Given the affordability, widespread use, and availability of computer graphics hardware, here we describe a computationally efficient GPU-based method that approximates acoustical occlusion/diffraction effects in real time. Although the method has been developed primarily for videogames where occlusion/diffraction is typically overlooked, it is relevant for dynamic and interactive virtual environments as well.

SCETF: Serious game surgical cognitive education and training framework

Surgical proficiency requires command of both technical and cognitive skills. Although at times overlooked, cognitive skills training allows residents to practise detecting errors ultimately leading to a reduction of errors. Virtual simulations and serious games offer a viable alternative to practice in an actual operating room where traditionally both technical and cognitive skills acquisition takes places. They provide residents the opportunity to train until they reach a specific competency level in a safe, cost effective, fun, and engaging manner allowing them to make more effective use of their limited training time in the operating room. Here we introduce a serious game surgical cognitive education and training framework (SCETF) that is currently being developed specifically for cognitive surgical skills training. Domain-specific surgical “modules” can then be built on top of the existing framework, utilizing common simulation elements/assets. The SCETF is being developed as a research tool where various simulation parameters such as levels of audio and visual fidelity, can be easily adjusted allowing for the controlled testing of such factors on knowledge transfer and retention.

A real-time reconstructed 3D environment augmented with virtual objects rendered with correct occlusion

In this work we present a novel framework for the real-time interaction with 3D models in augmented virtual reality. Our framework incorporates view-dependent stereoscopic rendering of the reconstructed environment together with users hands and a virtual object, and high-precision gesture recognition to manipulate it. Proposed setup consists of a Creative RGB-D camera, Oculus Rift VR head mounted display (HMD), Leap Motion hands and fingers tracker and an AR marker. The system is capable of augmenting the users hands in relation to their point of view (POV) using the depth sensor mounted on the HMD, and allows manipulation of the environment through the Leap Motion sensor. The AR marker is used to determine the location of the Leap Motion sensor to help with consolidation of transformations between the Oculus and the Leap Motion sensor. Combined with accurate information from the Oculus HMD, the system is able to track the users head and fingers, with 6-DOF, to provide a spatially accurate augmentation of the users virtual hands. Such an approach allows us to achieve high level of user immersion since the augmented objects occlude the users hands properly; something which is not possible with conventional AR. We hypothesize that users of our system will be able to perform better object manipulation tasks in this particular augmented VR setup as compared to virtual reality (VR) where users hands are not visible, or if visible, always occlude virtual objects.

Effects of sound on visual realism perception and task performance

Before the application of virtual simulations and serious games for surgical education and training becomes more widespread, there are a number of open questions and issues that must be addressed including the relationship between realism, multi-modal cue interaction, immersion, and knowledge transfer and retention. Using the serious game surgical cognitive education and training framework developed specifically for cognitive surgical skills training, here we examine the effect of sound on visual realism perception and task completion time while performing a task within a virtual environment. Our preliminary experimental results indicate that the appropriate use of sound can lead to performance improvements when performing a task within a virtual environment without a corresponding decrease in the perception of visual realism.

Interprofessional care simulator for critical care education

Interprofessional education is a pedagogical approach which allows health care practitioners to develop a clear understanding and appreciation of the roles, expertise, and unique contributions of their disciplines as well as those of the other participating health care providers. It also helps build effective team relationships which is essential for optimal health care delivery. Interactive virtual simulation environments, such as serious games, can be used to augment traditional classroom-based interprofessional teaching methods as multiple team members may participate in the simulation simultaneously regardless of their physical location or time of day. Here we describe the Interprofessional Care Simulator for interprofessional education.

Gamification and health professions education

Here we analyze and provide a description of our ongoing work that couples an online learning management system and gamification to provide an engaging and motivating learning experience for health professions students.

A GPU-Based Method to Approximate Acoustical Reflectivity

Abstract Here we present a GPU-based heuristic method for approximating the size and reflectivity of a room relative to a given sound source origin that can potentially provide interactive response for the appropriate selections of geometry and hardware. The method calculates the average distance to the nearest surfaces in the environment and an approximate measure of the fraction of sound energy that would be reflected from those surfaces. Both of these values are then used to drive the reverberation effects of an audio API. The method can potentially provide interactive response for the appropriate selections of geometry and hardware. Source code is available online at address provided at the end of this paper.

The effect of sound on visual realism perception and task completion time in a cel-shaded serious gaming virtual environment

Here we investigate the effect of sound on the perception of visual realism and the time required to complete a simple navigation-based task within a serious gaming (virtual) environment under various sound and visual conditions. Results indicate that the perception of visual realism and task completion time can be affected by sound. Designers and developers of serious games (and virtual environments in general) should be aware of the effects of sound on a users perception of the visual scene and on task completion time, and they should thus ensure that sound is carefully considered when creating such environments.