Ingrid Carlbom

A beam tracing approach to acoustic modeling for interactive virtual environments

Virtual environment research has focused on interactive image generation and has largely ignored acoustic modeling for spatialization of sound. Yet, realistic auditory cues can complement and enhance visual cues to aid navigation, comprehension, and sense of presence in virtual environments. A primary challenge in acoustic modeling is computation of reverberation paths from sound sources fast enough for real-time auralization. We have developed a system that uses precomputed spatial subdivision and “beam tree” data structures to enable real-time acoustic modeling and auralization in interactive virtual environments. The spatial subdivision is a partition of 3D space into convex polyhedral regions (cells) represented as a cell adjacency graph. A beam tracing algorithm recursively traces pyramidal beams through the spatial subdivision to construct a beam tree data structure representing the regions of space reachable by each potential sequence of transmission and specular reflection events at cell boundaries. From these precomputed data structures, we can generate high-order specular reflection and transmission paths at interactive rates to spatialize fixed sound sources in real-time as the user moves through a virtual environment. Unlike previous acoustic modeling work, our beam tracing method: 1) supports evaluation of reverberation paths at interactive rates, 2) scales to compute highorder reflections and large environments, and 3) extends naturally to compute paths of diffraction and diffuse reflection efficiently. We are using this system to develop interactive applications in which a user experiences a virtual environment immersively via simultaneous auralization and visualization.

Modeling acoustics in virtual environments using the uniform theory of diffraction

Realistic modeling of reverberant sound in 3D virtual worlds provides users with important cues for localizing sound sources and understanding spatial properties of the environment. Unfortunately, current geometric acoustic modeling systems do not accurately simulate reverberant sound. Instead, they model only direct transmission and specular reflection, while diffraction is either ignored or modeled through statistical approximation. However, diffraction is important for correct interpretation of acoustic environments, especially when the direct path between sound source and receiver is occluded. The Uniform Theory of Diffraction (UTD) extends geometrical acoustics with diffraction phenomena: illuminated edges become secondary sources of diffracted rays that in turn may propagate through the environment. In this paper, we propose an efficient way for computing the acoustical effect of diffraction paths using the UTD for deriving secondary diffracted rays and associated diffraction coefficients. Our main contributions are: 1) a beam tracing method for enumerating sequences of diffracting edges efficiently and without aliasing in densely occluded polyhedral environments; 2) a practical approximation to the simulated sound field in which diffraction is considered only in shadow regions; and 3) a real-time auralization system demonstrating that diffraction dramatically improves the quality of spatialized sound in virtual environments.

Planar Geometric Projections and Viewing Transformations

In computer graphics one is often concerned with representing three-dimensional objects on a two-dimensional display surface. The choice of such a representation depends on several factors, including the purpose for which the representation is intended, the visual effects tha t are desired, and the shape of the object. This paper describes how twodimensional views can be obtained using planar geometric projections such as perspective and parallel projections. It discusses how these projections can be generated from a threedimensional representation of an object in a manner suitable for computer graphics systems. In particular, it shows how these projections can be generated using the viewing transformations of the Core Graphics System. The factors tha t affect the choice of projection are also discussed, and some guidelines for making such a choice are given.

A Hierarchical Data Structure for Representing the Spatial Decomposition of 3-D Objects

The polytree, a generalization of the octree data structure, retains most of the desirable features of the octree structure while offering several advantages.

Real-time acoustic modeling for distributed virtual environments

Realistic acoustic modeling is essential for spatializing sound in distributed virtual environments where multiple networked users move around and interact visually and aurally in a shared virtual world. Unfortunately, current methods for computing accurate acoustical models are not fast enough for real-time auralization of sounds for simultaneously moving sources and receivers. In this paper, we present three new beam tracing algorithms that greatly accelerate computation of reverberation paths in a distributed virtual environment by taking advantage of the fact that sounds can only be generated or heard at the positions of “avatars” representing the users. The priority-driven beam tracing algorithm performs a bestfirst search of a cell adjacency graph, and thus enables new termination criteria with which all early reflection paths can be found very efficiently. The bidirectional beam tracing algorithm combines sets of beams traced from pairs of avatar locations to find reverberation paths between them while requiring significantly less computation than previous unidirectional algorithms. The amortized beam tracing algorithm computes beams emanating from box-shaped regions of space containing predicted avatar locations and re-uses those beams multiple times to compute reflections paths as each avatar moves inside the box. Cumulatively, these algorithms enable speedups of approximately two orders of magnitude over previous methods. They are incorporated into a time-critical multiprocessing system that allocates its computational resources dynamically in order to compute the highest priority reverberation paths between moving avatar locations in real-time with graceful degradation and adaptive refinement.

Real time tracking for enhanced tennis broadcasts

This paper develops real time tracking technology for sports broadcast applications. The specific sport chosen here is the game of tennis. The outputs of the tennis tracking system are spatio-temporal trajectories of motion of the players and the ball which can in turn provide a number of statistics about the game. For instance, the distance travelled by a player, the speed and the acceleration at any instant, as well as court coverage patterns can be obtained from the trajectories. The statistics so obtained can be visualized in compelling ways to enhance the appreciation of the athleticism and strategy involved in the sport. We present techniques for tracking the players and the ball in video obtained from stationary cameras. The problem is challenging as the tracking needs to be performed outdoors, players are fast-moving non-rigid objects, and the ball is a small object that can move at speeds in the range of 150 miles an hour. Player trajectories are obtained by dynamically clustering tracks of local features. Ball segmentation and tracking is based on shape and color features of the ball. Real time tracking results are presented on video recorded live by the authors in an international tennis tournament.

Plenoptic stitching: a scalable method for reconstructing 3D interactive walk throughs

Interactive walkthrough applications require detailed 3D models to give users a sense of immersion in an environment. Traditionally these models are built using computer-aided design tools to define geometry and material properties. But creating detailed models is time-consuming and it is also difficult to reproduce all geometric and photometric subtleties of real-world scenes. Computer vision attempts to alleviate this problem by extracting geometry and photogrammetry from images of the real-world scenes. However, these models are still limited in the amount of detail they recover. Image-based rendering generates novel views by resampling a set of images of the environment without relying upon an explicit geometric model. Current such techniques limit the size and shape of the environment, and they do not lend themselves to walkthrough applications. In this paper, we define a parameterization of the 4D plenoptic function that is particularly suitable for interactive walkthroughs and define a method for its sampling and reconstructing. Our main contributions are: 1) a parameterization of the 4D plenoptic function that supports walkthrough applications in large, arbitrarily shaped environments; 2) a simple and fast capture process for complex environments; and 3) an automatic algorithm for reconstruction of the plenoptic function.

Audio-visual tracking for natural interactivity

The goal in user interfaces is natural interactivity unencumbered by sensor and display technology. In this paper, we propose that a multi-modal approach using inverse modeling techniques from computer vision, speech recognition, and acoustics can result in such interfaces. In particular, we demonstrate a system for audio-visual tracking, showing that such a system is more robust, more accurate, more compact, and yields more information than using a single modality for tracking. We also demonstrate how such a system can be used to find the talker among a group of individuals, and render 3D scenes to the user.

Instantly indexed multimedia databases of real world events

We introduce a new paradigm for real-time conversion of a real world event into a rich multimedia database by processing data from multiple sensors observing the event. A real-time analysis of the sensor data, tightly coupled with domain knowledge, results in instant indexing of multimedia data at capture time. This yields semantic information to answer complex queries about the content and the ability to extract portions of data that correspond to complex actions performed in the real world. The power of such an instantly indexed multimedia database system, in content-based retrieval of multimedia data or in semantic analysis and visualization of the data, far exceeds that of systems which index multimedia data only after it is produced. We present LucentVision, an instantly indexed multimedia database system developed for the sport of tennis. This system analyzes video from multiple cameras in real time and captures the activity of the players and the ball in the form of motion trajectories. The system stores these trajectories in a database along with video, 3D models of the environment, scores, and other domain-specific information. LucentVision has been used to enhance live television and Internet broadcasts with game analyses and virtual replays in more than 250 international tennis matches.

An Algorithm for Geometric Set Operations Using Cellular Subdivision Techniques

Geometric set operations play an integral role in systems for CAD/CAM, for robot planning, and for modeling objects such as underground formations from empirical data. Two major issues in the implementation of geometric set operations are efficiency in the search for geometric intersections and effectiveness in the treatment of singular intersection cases. This article presents an algorithm for geometric set operations on planar polyhedral nonmanifold objects that addresses both these issues. First, an efficient search for geometric intersections is obtained by localizing the search to small regions of object space through a cellular subdivision scheme using the polytree data structure. Second, an effective treatment of singular intersection cases is obtained by mapping each singular intersection occurring in a region into one of a small set of cases.