Hannes Gamper

An augmented reality interface to contextual information

In this paper, we report on a prototype augmented reality (AR) platform for accessing abstract information in real-world pervasive computing environments. Using this platform, objects, people, and the environment serve as contextual channels to more information. The user’s interest with respect to the environment is inferred from eye movement patterns, speech, and other implicit feedback signals, and these data are used for information filtering. The results of proactive context-sensitive information retrieval are augmented onto the view of a handheld or head-mounted display or uttered as synthetic speech. The augmented information becomes part of the user’s context, and if the user shows interest in the AR content, the system detects this and provides progressively more information. In this paper, we describe the first use of the platform to develop a pilot application, Virtual Laboratory Guide, and early evaluation results of this application.

Analyzing emotional semantics of abstract art using low-level image features

In this work, we study peoples emotions evoked by viewing abstract art images based on traditional low-level image features within a binary classification framework. Abstract art is used here instead of artistic or photographic images because those contain contextual information that influences the emotional assessment in a highly individual manner. Whether an image of a cat or a mountain elicits a negative or positive response is subjective. After discussing challenges concerning image emotional semantics research, we empirically demonstrate that the emotions triggered by viewing abstract art images can be predicted with reasonable accuracy by machine using a variety of low-level image descriptors such as color, shape, and texture. The abstract art dataset that we created for this work has been made downloadable to the public.

GeoBoids: A mobile AR application for exergaming

We have designed a mobile Augmented Reality (AR) game which incorporates video see-through and spatialized audio AR techniques and encourages player movement in the real world. In the game, called GeoBoids, the player is surrounded by flocks of virtual creatures that are visible and audible through mobile AR application. The goal is for the player to run to the location of a GeoBoid swarm in the real world, capture all the creatures there, then run to the next swarm and repeat, before time runs out, encouraging the player to exercise during game play. The most novel elements of the game are the use of audio input and output for interacting with the creatures. The interface design of the game includes AR visualization, spatialized audio, touch gestures and whistle interaction. Feedback from users in a preliminary user study was mostly positive on overall game play and the design of the UI, while the results also revealed improvements were needed for whistle interaction and the visual design of the GeoBoids.

Assisted Listening Using a Headset: Enhancing audio perception in real, augmented, and virtual environments

Historically, headphones have mainly been used for analytic listening in music production and in homes. During the last decade, with the boom of dedicated music players and mobile phones, the everyday use of light headphones has become highly popular. Current headphones are also paving the way for more sophisticated assisted listening devices. Today, active noise control (ANC), equalization techniques, and a hear-through function are already a standard part of many headphones that people commonly use while traveling. It is not difficult to predict that, in the near future, a headset will be a ?hearing aid for those with normal hearing,? which can improve listening conditions for example in a noisy environment.

Anthropometric parameterisation of a spherical scatterer ITD model with arbitrary ear angles

Accurate modelling of the interaural time difference (ITD) is crucial for rendering localised sound. Parametric models allow personalising ITDs using anthropometrics. However, the mapping between anthropometric features and model parameters is not straightforward. Here, we propose deriving personalised ITD model parameters from a sphere fitted to a 3-D head scan. The proposed ITD personalisation is evaluated on an HRTF database containing 181 subjects, for a simple spherical ITD model as well as a frequency and elevation-dependent spherical scatterer model with arbitrary ear angles.

Synthesis of Device-Independent Noise Corpora for Realistic ASR Evaluation.

In order to effectively evaluate the accuracy of automatic speech recognition (ASR) with a novel capture device, it is important to create a realistic test data corpus that is representative of real-world noise conditions. Typically, this involves either recording the output of a device under test (DUT) in a noisy environment, or synthesizing an environment over loudspeakers in a way that simulates realistic signal-to-noise ratios (SNRs), reverberation times, and spatial noise distributions. Here we propose a method that aims at combining the realism of in-situ recordings with the convenience and repeatability of synthetic corpora. A device-independent spatial recording containing noise and speech is combined with the measured directivity pattern of a DUT to generate a synthetic test corpus for evaluating the performance of an ASR system. This is achieved by a spherical harmonic decomposition of both the sound field and the DUT’s directivity patterns. Experimental results suggest that the proposed method can be a viable alternative to costly and cumbersome device-dependent measurements. The proposed simulation method predicted the SNR of the DUT response to within about 3 dB and the word error rate (WER) to within about 20%, across a range of test SNRs, target source directions, and noise types.

Estimation of multipath propagation delays and interaural time differences from 3-D head scans

The estimation of acoustic propagation delays from a sound source to a listeners ear entrances is useful for understanding and visualising the wave propagation along the surface of the head, and necessary for individualised spatial sound rendering. The interaural time difference (ITD) is of particular research interest, as it constitutes one of the main localisation cues exploited by the human auditory system. Here, an approach is proposed that employs ray tracing on a 3-D head scan to estimate and visualise the propagation delays and ITDs from a sound source to a subjects ear entrances. Experimental results indicate that the proposed approach is computationally efficient, and performs equally well or better than optimally tuned parametric ITD models, with a mean absolute ITD estimation error of about 14μs.

Synthesis of device-independent noise corpora for speech quality assessment

The perceived quality of speech captured in the presence of background noise is an important performance metric for communication devices, including portable computers and mobile phones. For a realistic evaluation of speech quality, a device under test (DUT) needs to be exposed to a variety of noise conditions either in real noise environments or via noise recordings, typically delivered over a loudspeaker system. However, the test data obtained this way is specific to the DUT and needs to be re-recorded every time the DUT hardware changes. Here we propose an approach that uses device-independent spatial noise recordings to generate device-specific synthetic test data that simulate in-situ recordings. Noise captured using a spherical microphone array is combined with the directivity patterns of the DUT, referred to here as device-related transfer functions (DRTFs), in the spherical harmonics domain. The performance of the proposed method is evaluated in terms of the predicted signal-to-noise ratio (SNR) and the predicted mean opinion score (PMOS) of the DUT under various noise conditions. The root-mean-squared errors (RMSEs) of the predicted SNR and PMOS are on average below 4 dB and 0.28, respectively, across the range of tested SNRs, target source directions, noise types, and spherical harmonics decomposition methods. These experimental results indicate that the proposed method may be suitable for generating device-specific synthetic corpora from device-independent in-situ recordings.

Applications of 3D spherical transforms to personalization of head-related transfer functions

Head-related transfer functions (HRTFs) depend on the shape of the human head and ears, motivating HRTF personalization methods that detect and exploit morphological similarities between subjects in an HRTF database and a new user. Prior work determined similarity from sets of morphological parameters. Here we propose a non-parametric morphological similarity based on a harmonic expansion of head scans. Two 3D spherical transforms are explored for this task, and an appropriate shape similarity metric is defined. A case study focusing on personalisation of interaural time differences (ITDs) is conducted by applying this similarity metric on a database of 3D head scans.

Interaural time delay personalisation using incomplete head scans

When using a set of generic head-related transfer functions (HRTFs) for spatial sound rendering, personalisation can be considered to minimise localisation errors. This typically involves tuning the characteristics of the HRTFs or a parametric model according to the listeners anthropometry. However, measuring anthropometric features directly remains a challenge in practical applications, and the mapping between anthropometric and acoustic features is an open research problem. Here we propose matching a face template to a listeners head scan or depth image to extract anthropometric information. The deformation of the template is used to personalise the interaural time differences (ITDs) of a generic HRTF set. The proposed method is shown to outperform reference methods when used with high-resolution 3-D scans. Experiments with single-frame depth images indicate that the method is applicable to lower resolution or partial scans which are quicker and easier to obtain than full 3-D scans. These results suggest that the proposed method may be a viable option for ITD personalisation in practical applications.