Tobias Lentz

Virtual reality system with integrated sound field simulation and reproduction

A real-time audio rendering system is introduced which combines a full room-specific simulation, dynamic crosstalk cancellation, and multitrack binaural synthesis for virtual acoustical imaging. The system is applicable for any room shape (normal, long, flat, coupled), independent of the a priori assumption of a diffuse sound field. This provides the possibility of simulating indoor or outdoor spatially distributed, freely movable sources and a moving listener in virtual environments. In addition to that, near-to-head sources can be simulated by using measured near-field HRTFs. The reproduction component consists of a headphone-free reproduction by dynamic crosstalk cancellation. The focus of the project is mainly on the integration and interaction of all involved subsystems. It is demonstrated that the system is capable of real-time room simulation and reproduction and, thus, can be used as a reliable platform for further research on VR applications.

Processing of sound location in human cortex

This functional magnetic resonance imaging study was focused on the neural substrates underlying human auditory space perception. In order to present natural‐like sound locations to the subjects, acoustic stimuli convolved with individual head‐related transfer functions were used. Activation foci, as revealed by analyses of contrasts and interactions between sound locations, formed a complex network, including anterior and posterior regions of temporal lobe, posterior parietal cortex, dorsolateral prefrontal cortex and inferior frontal cortex. The distinct topography of this network was the result of different patterns of activation and deactivation, depending on sound location, in the respective voxels. These patterns suggested different levels of complexity in processing of auditory spatial information, starting with simple left/right discrimination in the regions surrounding the primary auditory cortex, while the integration of information on hemispace and eccentricity of sound may take place at later stages. Activations were identified as being located in regions assigned to both the dorsal and ventral auditory cortical streams, that are assumed to be preferably concerned with analysis of spatial and non‐spatial sound features, respectively. The finding of activations also in the ventral stream could, on the one hand, reflect the well‐known functional duality of auditory spectral analysis, that is, the concurrent extraction of information based on location (due to the spectrotemporal distortions caused by head and pinnae) and spectral characteristics of a sound source. On the other hand, this result may suggest the existence of shared neural networks, performing analyses of auditory ‘higher‐order’ cues for both localization and identification of sound sources.

Integrating real-time binaural acoustics into VR applications

Common research in the field of Virtual Reality (VR) considers acoustic stimulation as a highly important necessity for enhanced immersion into virtual scenes. However, most common VR toolkits do only marginally support the integration of sound for the application programmer. Furthermore, the quality of stimulation that is provided usually ranges from system sounds (e.g. beeps while selecting a menu) to simple 3D panning. In the latter case, these approaches do only allow the user to correctly detect sounds that are at quite a distance from his current position. Binaural synthesis is an interesting way to allow the spatial auditory representation by using few loudspeakers or headphones. This paper describes a system that combines the efforts of creating a binaural representation for the listener who is interacting in a common visual VR application in real-time, thus allowing the research on interaction between visual and auditory human perception systems. It will describe the theoretical background to establishing a binaural representation of a sound and the necessary hardware set-up for this. Afterwards, the infrastructure and software interface which will allow the connection of the audio renderer to a visual VR toolkit is discussed.

A true spatial sound system for CAVE-like displays using four loudspeakers

The paper introduces an audio rendering system based on the binaural approach, which allows a real-time simulation of spatially distributed sound sources and, in addition to that, near-to-head sources in room-mounted virtual environments. We have been developing a dynamic crosstalk cancellation, allowing the listener to freely move around without wearing any headphones. The paper gives a comprehensive description of the system, concentrating on the dual dynamic crosstalk cancellation and aspects of the integration of a real-time room acoustical simulation. Finally, two applications are described to show the wide applicability of the system.

Binaural acoustics for CAVE-like environments without headphones

The human auditory system, in contrast to the human visual system, can perceive input from all directions and has no limited field of view. As such, it provides valuable cues for navigation and orientation in virtual environments. However, audio stimuli are not that common in todays Virtual Reality applications, and this might result from the lack of middleware or user acceptance due to the need for specialized or costly hardware. Surprisingly, the lack of headphone-less near body acoustics is widely accepted, and simple intensity panning approaches that enable plausible spatial audio are used. This paper describes a networked environment for sophisticated binaural synthesis-based audio rendering in visual VR applications for a freely moving listener in a CAVE-like environment without the use of headphones. It describes the binaural acoustics rendering technique and a dynamic crosstalk cancellation system for four loudspeakers. In addition to that, synchronization issues and network coupling together with performance measurements that proof the applicability of the system in interactive Virtual Environments are discussed.