Jan Alexis Daniel Nesvadba

Evolvable visual commercial detector

Commercial detection plays an important role in various video segmentation and indexing applications. It provides high-level program segmentation so that other algorithms can be applied on the true program material in the broadcast. It is a challenge to have robust commercial detection methodology for various platforms, content formats, and broadcast styles that are used all over the world. Wide deployment of such an algorithm not only requires the development of new algorithms but also updating and tuning of parameters for existing algorithms. We present visual commercial detectors that rely on features including, luminance, letterbox, and keyframe distance. These detectors were developed after a careful study of the various features that can be extracted during MPEG-encoding process in real time. Due to the intermittent nature of the features, and platform restrictions, the commercial detection relies on a set of thresholds to keep the implementation as simple as possible. We evolved these thresholds using genetic algorithms (GAs) to optimize the performance. We show how a scalar genetic algorithm can locate sets of parameters in a multi-objective space (precision and recall) that outperform the values selected by an expert engineer. We present the results of optimizing a commercial detection algorithm for different data sets and parameter sets. In this paper we show that GAs drastically improved our approach and enabled fast prototyping and performance tuning of commercial detection algorithms.

Real-Time and Distributed AV Content Analysis System for Consumer Electronics Networks

The ever-increasing complexity of generic multimedia-content-analysis-based (MCA) solutions, their processing power demanding nature and the need to prototype and assess solutions in a fast and cost-saving manner motivated the development of the Cassandra framework. The combination of state-of-the-art network and grid-computing solutions and recently standardized interfaces facilitated the set-up of this framework, forming the basis for multiple cross-domain and cross-organizational collaborations. It enables distributed computing scenario simulations for e.g. distributed content analysis (DCA) across consumer electronics (CE) in-home networks, but also the rapid development and assessment of complex multi-MCA-algorithm-based applications and system solutions. Furthermore, the frameworks modular nature-logical MCA units are wrapped into so-called service units (SU)-ease the split between system-architecture- and algorithmic-related work and additionally facilitate reusability, extensibility and upgrade ability of those SUs

Comparison of shot boundary detectors

A video cut detector (CD), a member of the shot boundary detector (SBD) group, is an essential element for spatio-temporal audiovisual (AV) segmentation and various video-processing technologies. Platform, processing and performance constraints forced the development of various dedicated CDs. Future platforms allow the usage of advanced CD algorithms with higher reliability. In order to enable an appropriate trade-off decision to be made between reliability and the required processing power, benchmarking of four CD algorithms has taken place on bases of a generic, culture-diverse multi-genre AV corpus. In terms of complexity/performance trade-off, a field-difference-based CD proved to be optimal.

CANDELA – storage, analysis and retrieval of video content in distributed systems

Although many different types of technologies for information systems have evolved over the last decades (such as databases, video systems, the Internet and mobile telecommunication), the integration of these technologies is just in its infancy and has the potential to introduce ”intelligent” systems. The CANDELA project, which is part of the European ITEA program, focuses on the integration of video content analysis in combination with networked delivery and storage technologies. To unleash the full potential of such integration, adaptive video-content analysis and retrieval techniques are being explored by developing several pilot applications.

The color browser: a content driven linear video browsing tool

The proliferation of multimedia information sources and the availability of large capacity consumer storage devices, raise the need to present video in compact forms so that users can effectively access interesting parts to watch. Aiming at an implementation in consumer electronics products and targeting to non-IT-expert users, an innovative user-friendly tool for video browsing and retrieval has been developed. The color browser enhances conventional slide-bars by embodying information about the video content in its colored background. It exploits low level features, like the dominant colors or the volume of the sound-track, automatically extracted from the digital video streams to allow intuitive access and content-driven navigation through unstructured video sequences.

Face tracking in the compressed domain

A compressed domain generic object tracking algorithm offers, in combination with a face detection algorithm, a low-compu-tational-cost solution to the problem of detecting and locating faces in frames of compressed video sequences (such as MPEG-1 or MPEG-2). Objects such as faces can thus be tracked through a compressed video stream using motion information provided by existing forward and backward motion vectors. The described solution requires only low computational resources on CE devices and offers at one and the same time sufficiently good location rates.

CASSANDRA Framework: A Service Oriented Distributed Multimedia Content Analysis Engine

Connected Consumer Electronics (CE) devices face an exponential growth in storage, processing capabilities and connectivity bandwidth. This increased complexity calls for new software architectures that naturally admit self-organization, resource management for efficient workload distribution, and a transparent cooperation of connected CE-terminals. The interconnected nature makes advanced applications feasible such as the smart distribution of complex multimedia content analysis algorithms, resulting in automatic semantic content-awareness creation. In this paper we describe a modular, self-aware, self-organizing, real-time and distributed multimedia content analysis system. Based on a Service Oriented Architecture it is, furthermore, capable of efficiently and dynamically using the available resources in a grid-like manner.

Early evaluation of future consumer AV content analysis applications with PC networks

The paper deals with software productivity improvement for consumer multimedia devices by means of PC and component technology and shows how this is done for complex real-time content analysis applications used in advanced new storage products of the future. Content analysis is a relatively new and immature technology. It is used for browsing and searching particular content items among thousands of others on “big” embedded storage devices like hard disks. As the storage capacity of hard disk and flash continues to grow rapidly, content analysis is bound to become a key enabling technology in future storage products. A major problem with content analysis features (and many other features as well) is that underlying algorithms are unstable, sometimes unavailable, or at least, very much in their infancy, and as such, subject to frequent changes. The paper describes an approach to facilitate early evaluation and integration of such immature features. This is done by packing each feature, as-is, into components and by providing PC network technology to interconnect them. In our prototyping framework, each component is an independent executable program that runs on some PC in the network, streaming AV data via TCP/IP and being controlled through UPnP networking. Experiences with large-scale prototyping activities we have carried out for the assessment of future content analysis systems, show that a PC based prototyping approach enables the integration of many different media processing features in a short time and that it allows for accurate analysis of the resource (CPU/memory) requirements of such components.

Dialog detection in narrative video by shot and face analysis

The proliferation of captured personal and broadcast content in personal consumer archives necessitates comfortable access to stored audiovisual content. Intuitive retrieval and navigation solutions require however a semantic level that cannot be reached by generic multimedia content analysis alone. A fusion with film grammar rules can help to boost the reliability significantly. The current paper describes the fusion of low-level content analysis cues including face parameters and inter-shot similarities to segment commercial content into film grammar rule-based entities and subsequently classify those sequences into so-called shot reverse shots, i.e. dialog sequences. Moreover shot reverse shot specific mid-level cues are analyzed augmenting the shot reverse shot information with dialog specific descriptions.

Distributed and adaptive multimedia content analysis prototyping framework for consumer electronics

The ever-increasing complexity of generic multimedia-content-analysis-based (MCA) solutions, their processing power demanding nature and the need to prototype and assess solutions in a fast and cost-saving manner motivated the development of the Cassandra framework. The combination of state-of-the-art network and grid-computing solutions and recently standardized interfaces facilitated the set-up of this framework, forming the basis for multiple cross-domain and cross-organizational collaborations. It enables distributed computing scenario simulations for, for example, distributed content analysis (DCA) across consumer electronics (CE) in-home networks, but also the rapid development and assessment of complex multi-MCA-algorithm-based applications and system solutions. Furthermore, the frameworks modular nature - logical MCA units are wrapped into so-called service units (SU) - ease the split between system-architecture - and algorithmic-related work and additionally facilitate reusability, extensibility and upgradeability of those SUs