Helmut Neuschmied

Content-based Video Retrieval and Summarization using MPEG-7

Retrieval in current multimedia databases is usually limited to browsing and searching based on low-level visual features and explicit textual descriptors. Semantic aspects of visual information are mainly described in full text attributes or mapped onto specialized, application specific description schemes. Result lists of queries are commonly represented by textual descriptions and single key frames. This approach is valid for text documents and images, but is often insufficient to represent video content in a meaningful way. In this paper we present a multimedia retrieval framework focusing on video objects, which fully relies on the MPEG-7 standard as information base. It provides a content-based retrieval interface which uses hierarchical content-based video summaries to allow for quick viewing and browsing through search results even on bandwidth limited Web applications. Additionally semantic meaning about video content can be annotated based on domain specific ontologies, enabling a more targeted search for content. Our experiences and results with these techniques will be discussed in this paper.

Content-based identification of audio titles on the Internet

The increasing usage of the Internet for the distribution of audio content and the increasing number of audio broadcasting stations require new supporting tools and methods for the observation of occurrence frequencies and of possible copyright infringements. The paper describes a general approach for the identification of audio titles and its application on Internet observation. The concept of an AudioDNA is developed, allowing a highly compressed representation of a sequence of acoustic events. Audio titles are identified by using a sequence matching method which determines similarities between observed and reference AudioDNA stored in a database. This method is implemented in a highly scaleable architecture allowing the identification of several audio titles in parallel. The first results with this system are very promising, only highly distorted audio titles are not identified correctly.

Smartphone eye tracking toolbox: accurate gaze recovery on mobile displays

Human Interaction with mobile devices has recently been estab-lished as application field in eye tracking research. Current technologies for gaze recovery on mobile displays cannot enable fully natural interaction with the mobile device: users are condi-tioned to interact with tightly mounted displays or distracted by markers in their view. We propose a novel approach that cap-tures point-of-regards (PORs) with eye tracking glasses (ETG) and then uses computer vision methodology for the robust local-ization of the smartphone in the head camera video. We present an integrated software package, i.e., the Smartphone Eye Track-ing Toolbox (SMET) that enables accurate gaze recovery on mobile displays with heat mapping of recent attention. We re-port the performance of the computer vision approach and demonstrate it with various natural interaction scenarios using the SMET Toolbox, enable ROI settings on the mobile display and show results from eye movement analysis, such as, ROI dwell time and statistics on eye gaze event (saccades, fixations).

Fast annotation of video objects for interactive TV

In this demonstration, we present the Annotation Tool that is being developed in the porTiVity project to annotate video objects for Interactive Television programs. This tool includes various video processing components to structure and speed-up the annotation process, such as shot segmentation, key-frame extraction, object tracking and object redetection. A specific feature is that the tool includes a preprocessing phase where a quantity of information is precomputed, so that the annotation itself can be done quite rapidly.

Recognition and analysis of audio for copyright protection: the RAA project

Automatic generation of play lists for commercial broadcast radio stations has become a major research topic. Audio identification systems have been around for a while, and they show good performance for clean audio files. However, songs transmitted by commercial radio stations are highly distorted to cause greater impact on the casual listener. This impact helps increase the probability that the listener will stay tuned in, but the price we have to pay is a severe modification in the audio itself. This causes the failure of traditional identification systems. Another problem is the fact that songs are never played from the beginning to the end. Actually, they are put on the air several seconds after their real beginning and almost always under the voice of a speaker. The same thing happens at the end. In this article, we present the RAA project, which was conceived to deal with real broadcast audio problems. The idea behind this project is to extract automatically an audio fingerprint (the so-called AudioDNA) that identifies the fragment of audio. This AudioDNA has to be robust enough to appear almost the same under several degrees of distortion. Once this AudioDNA is extracted from the broadcast audio, a matching algorithm is able to find its fragments inside a database. With this approach, the system can find not only a whole song but also small fragments of it, even with high distortion caused by broadcast (and DJ) manipulations.

A Cross Media Platform for Personalized Leisure & Entertainment: The POLYMNIA Approach

The POLYMNIA project aims to develop an intelligent cross-media platform for personalised leisure and entertainment in thematic parks or venues. The system allows the visitors to be the real protagonist in the venue. Towards this goal, POLYMNIA platform is equipped with innovative imaging technologies for real time detection, localisation and tracking of i human contenti, i.e., the human visitor within the recoding being made in real-time by the system. No constraints are imposed on the variation of the environment. New, content-based media representation and organisation schemes will be developed to provide scalable, efficient and user-oriented description of the i human contenti, enabling efficient retrieval, access, and delivery across heterogeneous media platforms. In addition, adaptive mechanisms are employed to update the system response to the current usersi information needs and preferences

Multimodal Semantic Analysis of Public Transport Movements

We present a system for multimodal, semantic analysis of person movements that incorporates data from surveillance cameras, weather sensors, and third-party information providers. The interactive demonstration will show the automated creation of a survey of passenger transfer behavior at a public transport hub. Such information is vital for public transportation planning and the presented approach increases the cost-effectiveness and data accuracy as compared to traditional methods.

Attention in mobile interactions: gaze recovery for large scale studies

Understanding human attention in mobile interaction is a relevant part of human computer interaction, indica-ting focus of task, emotion and communication. Lack of large scale studies enabling statistically significant re-sults is due to high costs of manual penetration in eye tracking analysis. With high quality wearable cameras for eye-tracking and Google glasses, video analysis for visual attention analysis will become ubiquitous for automated large scale annotation. We describe for the first time precise gaze estimation on mobile displays and surrounding, its performance and without markers. We demonstrate accurate POR (point of regard) re-covery on the mobile device and enable heat mapping of visual tasks. In a benchmark test we achieve a mean accuracy in the POR localization on the display by 1.5 mm, and the method is very robust to illumination changes. We conclude from these results that this sys-tem may open new avenues in eye tracking research for behavior analysis in mobile applications.

Active Objects in Interactive Mobile TV

The porTiVity project is developing a converged rich media iTV system, which integrates broadcast and mobile broadband delivery to portables and mobiles and which will enable the end-user to act on moving objects within TV programmes. The developments of the project include the playout of portable rich media iTV and the middleware, data and presentation engine in the handheld receiver. In this demonstration, we will present, on the production side, the Live Annotation Tool which allows the video editor to define and include active objects in Live TV Programs, and on the user side, the interaction with active objects on a mobile terminal.

Near-Miss Accidents – Classification and Automatic Detection

In this work, we propose a system that automatically identifies hazardous traffic situations in order to gather comprehensive evidence, allowing timely mitigation of dangerous traffic areas. The system employs optical and acoustic sensors, stores the recorded sensor data to an incident store, and provides an assessment of the causes and consequences of the captured situation. Three main categories of features are used to assess the risk of a traffic situation: (1) key parameters of the traffic participants such as size, their distance, acceleration and motion trajectories; (2) the occurrence of acoustic events (shouting, tire squealing, honking sounds, etc.) which often co-occur with hazardous situations; (3) global parameters which describe the current traffic situation, such as traffic volume or density. An automated detection allows to monitor an intersection for an extensive time period. Compared to traditional manual methods, this facilitates generating significantly more data, which increases the informative value of such an assessment and therefore leads to a better understanding of the hazard potential of the spot. The outcome of such an investigation will finally serve as a basis for defining and prioritizing improvements.